Answer the following questions to see if you can safely skip this chapter. You can find the answers at the end of the chapter in Section 2.7.

1. Given the following data frame, how do I create a new column called “3” that contains the sum of 1 and 2? You may only use $, not [[. What makes 1, 2, and 3 challenging as variable names?

   df <- data.frame(runif(3), runif(3))
   names(df) <- c(1, 2)
   

2. In the following code, how much memory does y occupy?

   x <- runif(1e6)
   y <- list(x, x, x)
   

3. On which line does a get copied in the following example?

   a <- c(1, 5, 3, 2)
   b <- a
   b[[1]] <- 10
   

• Section 2.2 introduces you to the distinction between names and values, and discusses how <- creates a binding, or reference, between a name and a value.

• Section 2.3 describes when R makes a copy: whenever you modify a vector, you’re almost certainly creating a new, modified vector. You’ll learn how to use tracemem() to figure out when a copy actually occurs. Then you’ll explore the implications as they apply to function calls, lists, data frames, and character vectors.

• Section 2.4 explores the implications of the previous two sections on how much memory an object occupies. Since your intuition may be profoundly wrong and since utils::object.size() is unfortunately inaccurate, you’ll learn how to use lobstr::obj_size().

• Section 2.5 describes the two important exceptions to copy-on-modify: with environments and values with a single name, objects are actually modified in place.

• Section 2.6 concludes the chapter with a discussion of the garbage collector, which frees up the memory used by objects no longer referenced by a name.

We’ll use the lobstr (https://github.com/r-lib/lobstr) package to dig into the internal representation of R objects.

library(lobstr)

The details of R’s memory management are not documented in a single place. Much of the information in this chapter was gleaned from a close reading of the documentation (particularly ?Memory and ?gc), the memory profiling (http://cran.r-project.org/doc/manuals/R-exts.html#Profiling-R-code-for-memory-use) section of Writing R extensions [R Core Team, 2018a], and the SEXPs (http://cran.r-project.org/doc/manuals/R-ints.html#SEXPs) section of R internals [R Core Team, 2018b]. The rest I figured out by reading the C source code, performing small experiments, and asking questions on R-devel. Any mistakes are entirely mine.

R has strict rules about what constitutes a valid name. A syntactic name must consist of letters¹, digits, . and _ but can’t begin with _ or a digit. Additionally, you can’t use any of the reserved words like TRUE, NULL, if, and function (see the complete list in ?Reserved). A name that doesn’t follow these rules is a non-syntactic name; if you try to use them, you’ll get an error:

_abc <- 1
#> Error: unexpected input in "_"

if <- 10
#> Error: unexpected assignment in "if <-"

It’s possible to override these rules and use any name, i.e., any sequence of characters, by surrounding it with backticks:

`_abc` <- 1
`_abc`
#> [1] 1

`if` <- 10
`if`
#> [1] 10

While it’s unlikely you’d deliberately create such crazy names, you need to understand how these crazy names work because you’ll come across them, most commonly when you load data that has been created outside of R.

1. Explain the relationship between a, b, c and d in the following code:

   a <- 1:10
   b <- a
   c <- b
   d <- 1:10
   

2. The following code accesses the mean function in multiple ways. Do they all point to the same underlying function object? Verify this with lobstr::obj_addr().

   mean
   base::mean
   get("mean")
   evalq(mean)
   match.fun("mean")
   

3. By default, base R data import functions, like read.csv(), will automatically convert non-syntactic names to syntactic ones. Why might this be problematic? What option allows you to suppress this behaviour

4. What rules does make.names() use to convert non-syntactic names into syntactic ones?

5. I slightly simplified the rules that govern syntactic names. Why is .123e1 not a syntactic name? Read ?make.names for the full details.

You can see when an object gets copied with the help of base::tracemem(). Once you call that function with an object, you’ll get the object’s current address:

x <- c(1, 2, 3)
cat(tracemem(x), "\n")
#> <0x7f80c0e0ffc8>

From then on, whenever that object is copied, tracemem() will print a message telling you which object was copied, its new address, and the sequence of calls that led to the copy:

y <- x
y[[3]] <- 4L
#> tracemem[0x7f80c0e0ffc8 -> 0x7f80c4427f40]:

If you modify y again, it won’t get copied. That’s because the new object now only has a single name bound to it, so R applies modify-in-place optimisation. We’ll come back to this in Section 2.5.

y[[3]] <- 5L

untracemem(y)

untracemem() is the opposite of tracemem(); it turns tracing off.

The same rules for copying also apply to function calls. Take this code:

f <- function(a) {
  a
}

x <- c(1, 2, 3)
cat(tracemem(x), "\n")
#> <0x7f8e10fd5d38>

z <- f(x)
# there's no copy here!

untracemem(x)

While f() is running, the a inside the function points to the same value as the x does outside the function:

You’ll learn more about the conventions used in this diagram in Section 7.4.4. In brief: the function f() is depicted by the yellow object on the right. It has a formal argument, a, which becomes a binding (indicated by dotted black line) in the execution environment (the gray box) when the function is run.

Once f() completes, x and z will point to the same object. 0x74b never gets copied because it never gets modified. If f() did modify x, R would create a new copy, and then z would bind that object.

It’s not just names (i.e. variables) that point to values; elements of lists do too. Consider this list, which is superficially very similar to the numeric vector above:

l1 <- list(1, 2, 3)

This list is more complex because instead of storing the values itself, it stores references to them:

This is particularly important when we modify a list:

l2 <- l1

l2[[3]] <- 4

Like vectors, lists use copy-on-modify behaviour; the original list is left unchanged, and R creates a modified copy. This, however, is a shallow copy: the list object and its bindings are copied, but the values pointed to by the bindings are not. The opposite of a shallow copy is a deep copy where the contents of every reference are copied. Prior to R 3.1.0, copies were always deep copies.

To see values that are shared across lists, use lobstr::ref(). ref() prints the memory address of each object, along with a local ID so that you can easily cross-reference shared components.

ref(l1, l2)
#> ❚ [1:0x7f8e161c9198] <list>
#> ❘—[2:0x7f8e1689c178] <dbl>
#> ❘—[3:0x7f8e1689c140] <dbl>
#> ⃓—[4:0x7f8e1689c108] <dbl>
#>
#> ❚ [5:0x7f8e162d0c98] <list>
#> ❘—[2:0x7f8e1689c178]
#> ❘—[3:0x7f8e1689c140]
#> ⃓—[6:0x7f8e163423a0] <dbl>

Data frames are lists of vectors, so copy-on-modify has important consequences when you modify a data frame. Take this data frame as an example:

d1 <- data.frame(x = c(1, 5, 6), y = c(2, 4, 3))

If you modify a column, only that column needs to be modified; the others will still point to their original references:

d2 <- d1
d2[, 2] <- d2[, 2] * 2

However, if you modify a row, every column is modified, which means every column must be copied:

d3 <- d1
d3[1, ] <- d3[1, ] * 3

The final place that R uses references is with character vectors³. I usually draw character vectors like this:

x <- c("a", "a", "abc", "d")

But this is a polite fiction. R actually uses a global string pool where each element of a character vector is a pointer to a unique string in the pool:

You can request that ref() show these references by setting the character argument to TRUE:

ref(x, character = TRUE)
#> ❚ [1:0x7f8e149ef008] <chr>
#> ❘—[2:0x7f8e1094d598] <string: "a">
#> ❘—[2:0x7f8e1094d598]
#> ❘—[3:0x7f8e152e21c8] <string: "abc">
#> ⃓—[4:0x7f8e10b0e6b0] <string: "d">

This has a profound impact on the amount of memory a character vector uses but is otherwise generally unimportant, so elsewhere in the book I’ll draw character vectors as if the strings lived inside a vector.

1. Why is tracemem(1:10) not useful?

2. Explain why tracemem() shows two copies when you run this code. Hint: carefully look at the difference between this code and the code shown earlier in the section.

   x <- c(1L, 2L, 3L)
   tracemem(x)
   
   x[[3]] <- 4
   

3. Sketch out the relationship between the following objects:

   a <- 1:10
   b <- list(a, a)
   c <- list(b, a, 1:10)
   

4. What happens when you run this code?

   x <- list(1:10)
   x[[2]] <- x
   

   Draw a picture.

1. In the following example, why are object.size(y) and obj_size(y) so radically different? Consult the documentation of object.size().

   y <- rep(list(runif(1e4)), 100)
   
   object.size(y)
   #> 8005648 bytes
   obj_size(y)
   #> 80,896 B
   

2. Take the following list. Why is its size somewhat misleading?

   funs <- list(mean, sd, var)
   obj_size(funs)
   #> 17,608 B
   

3. Predict the output of the following code:

   a <- runif(1e6)
   obj_size(a)
   
   b <- list(a, a)
   obj_size(b)
   obj_size(a, b)
   
   b[[1]][[1]] <- 10
   obj_size(b)
   obj_size(a, b)
   
   b[[2]][[1]] <- 10
   obj_size(b)
   obj_size(a, b)
   

If an object has a single name bound to it, R will modify it in place:

v <- c(1, 2, 3)

v[[3]] <- 4

(Note the object IDs here: v continues to bind to the same object, 0x207.)

Two complications make predicting exactly when R applies this optimisation challenging:

• When it comes to bindings, R can currently⁶ only count 0, 1, or many. That means that if an object has two bindings, and one goes away, the reference count does not go back to 1: one less than many is still many. In turn, this means that R will make copies when it sometimes doesn’t need to.

• Whenever you call the vast majority of functions, it makes a reference to the object. The only exception are specially written “primitive” C functions. These can only be written by R-core and occur mostly in the base package.

Together, these two complications make it hard to predict whether or not a copy will occur. Instead, it’s better to determine it empirically with tracemem().

Let’s explore the subtleties with a case study using for loops. For loops have a reputation for being slow in R, but often that slowness is caused by every iteration of the loop creating a copy. Consider the following code. It subtracts the median from each column of a large data frame:

x <- data.frame(matrix(runif(5 * 1e4), ncol = 5))
medians <- vapply(x, median, numeric(1))

for (i in seq_along(medians)) {
  x[[i]] <- x[[i]] - medians[[i]]
}

This loop is surprisingly slow because each iteration of the loop copies the data frame. You can see this by using tracemem():

cat(tracemem(x), "\n")
#> <0x7f80c429e020>

for (i in 1:5) {
  x[[i]] <- x[[i]] - medians[[i]]
}
#> tracemem[0x7f80c429e020 -> 0x7f80c0c144d8]:
#> tracemem[0x7f80c0c144d8 -> 0x7f80c0c14540]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c14540 -> 0x7f80c0c145a8]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c145a8 -> 0x7f80c0c14610]:
#> tracemem[0x7f80c0c14610 -> 0x7f80c0c14678]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c14678 -> 0x7f80c0c146e0]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c146e0 -> 0x7f80c0c14748]:
#> tracemem[0x7f80c0c14748 -> 0x7f80c0c147b0]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c147b0 -> 0x7f80c0c14818]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c14818 -> 0x7f80c0c14880]:
#> tracemem[0x7f80c0c14880 -> 0x7f80c0c148e8]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c148e8 -> 0x7f80c0c14950]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c14950 -> 0x7f80c0c149b8]:
#> tracemem[0x7f80c0c149b8 -> 0x7f80c0c14a20]: [[<-.data.frame [[<-
#> tracemem[0x7f80c0c14a20 -> 0x7f80c0c14a88]: [[<-.data.frame [[<-

untracemem(x)

In fact, each iteration copies the data frame not once, not twice, but three times! Two copies are made by [[.data.frame, and a further copy⁷ is made because [[.data.frame is a regular function that increments the reference count of x.

We can reduce the number of copies by using a list instead of a data frame. Modifying a list uses internal C code, so the references are not incremented and only a single copy is made:

y <- as.list(x)
cat(tracemem(y), "\n")
#> <0x7f80c5c3de20>

for (i in 1:5) {
  y[[i]] <- y[[i]] - medians[[i]]
}
#> tracemem[0x7f80c5c3de20 -> 0x7f80c48de210]:

While it’s not hard to determine when a copy is made, it is hard to prevent it. If you find yourself resorting to exotic tricks to avoid copies, it may be time to rewrite your function in C++, as described in Chapter 25.

You’ll learn more about environments in Chapter 7, but it’s important to mention them here because their behaviour is different from that of other objects: environments are always modified in place. This property is sometimes described as reference semantics because when you modify an environment all existing bindings to that environment continue to have the same reference.

Take this environment, which we bind to e1 and e2:

e1 <- rlang::env(a = 1, b = 2, c = 3)
e2 <- e1

If we change a binding, the environment is modified in place:

e1$c <- 4
e2$c
#> [1] 4

This basic idea can be used to create functions that “remember” their previous state. See Section 10.2.4 for more details. This property is also used to implement the R6 object-oriented programming system, the topic of Chapter 14.

One consequence of this is that environments can contain themselves:

e <- rlang::env()
e$self <- e

ref(e)
#> ❚ [1:0x7f8e18918480] <env>
#> ⃓—self = [1:0x7f8e18918480]

This is a unique property of environments!

1. Explain why the following code doesn’t create a circular list.

   x <- list()
   x[[1]] <- x
   

2. Wrap the two methods for subtracting medians into two functions, then use the ‘bench’ package [Hester, 2018] to carefully compare their speeds. How does performance change as the number of columns increase?

3. What happens if you attempt to use tracemem() on an environment?

Take this short quiz to determine if you need to read this chapter. If the answers quickly come to mind, you can comfortably skip this chapter. You can check your answers in Section 3.8.

1. What are the four common types of atomic vectors? What are the two rare types?

2. What are attributes? How do you get them and set them?

3. How is a list different from an atomic vector? How is a matrix different from a data frame?

4. Can you have a list that is a matrix? Can a data frame have a column that is a matrix?

5. How do tibbles behave differently from data frames?

• Section 3.2 introduces you to the atomic vectors: logical, integer, double, and character. These are R’s simplest data structures.

• Section 3.3 takes a small detour to discuss attributes, R’s flexible metadata specification. The most important attributes are names, dimensions, and class.

• Section 3.4 discusses the important vector types that are built by combining atomic vectors with special attributes. These include factors, dates, date-times, and durations.

• Section 3.5 dives into lists. Lists are very similar to atomic vectors, but have one key difference: an element of a list can be any data type, including another list. This makes them suitable for representing hierarchical data.

• Section 3.6 teaches you about data frames and tibbles, which are used to represent rectangular data. They combine the behaviour of lists and matrices to make a structure ideally suited for the needs of statistical data.

Each of the four primary types has a special syntax to create an individual value, AKA a scalar⁴:

• Logicals can be written in full (TRUE or FALSE), or abbreviated (T or F).

• Doubles can be specified in decimal (0.1234), scientific (1.23e4), or hexadecimal (0xcafe) form. There are three special values unique to doubles: Inf, -Inf, and NaN (not a number). These are special values defined by the floating point standard.

• Integers are written similarly to doubles but must be followed by L⁵ (1234L, 1e4L, or 0xcafeL), and can not contain fractional values.

• Strings are surrounded by ” (”hi”) or ’ (’bye’). Special characters are escaped with \; see ?Quotes for full details.

To create longer vectors from shorter ones, use c(), short for combine:

lgl_var <- c(TRUE, FALSE)
int_var <- c(1L, 6L, 10L)
dbl_var <- c(1, 2.5, 4.5)
chr_var <- c("these are", "some strings")

When the inputs are atomic vectors, c() always creates another atomic vector; i.e. it flattens:

c(c(1, 2), c(3, 4))
#> [1] 1 2 3 4

In diagrams, I’ll depict vectors as connected rectangles, so the above code could be drawn as follows:

You can determine the type of a vector with typeof()⁶ and its length with length().

typeof(lgl_var)
#> [1] "logical"
typeof(int_var)
#> [1] "integer"
typeof(dbl_var)
#> [1] "double"
typeof(chr_var)
#> [1] "character"

R represents missing, or unknown values, with special sentinel value: NA (short for not applicable). Missing values tend to be infectious: most computations involving a missing value will return another missing value.

NA > 5
#> [1] NA
10 * NA
#> [1] NA
!NA
#> [1] NA

There are only a few exceptions to this rule. These occur when some identity holds for all possible inputs:

NA ^ 0
#> [1] 1
NA | TRUE
#> [1] TRUE
NA & FALSE
#> [1] FALSE

Propagation of missingness leads to a common mistake when determining which values in a vector are missing:

x <- c(NA, 5, NA, 10)
x == NA
#> [1] NA NA NA NA

This result is correct (if a little surprising) because there’s no reason to believe that one missing value has the same value as another. Instead, use is.na() to test for the presence of missingness:

is.na(x)
#> [1] TRUE FALSE   TRUE FALSE

NB: Technically there are four missing values, one for each of the atomic types: NA (logical), NA_integer_ (integer), NA_real_ (double), and NA_character_ (character). This distinction is usually unimportant because NA will be automatically coerced to the correct type when needed.

Generally, you can test if a vector is of a given type with an is.*() function, but these functions need to be used with care. is.logical(), is.integer(), is.double(), and is.character() do what you might expect: they test if a vector is a character, double, integer, or logical. Avoid is.vector(), is.atomic(), and is.numeric(): they don’t test if you have a vector, atomic vector, or numeric vector; you’ll need to carefully read the documentation to figure out what they actually do.

For atomic vectors, type is a property of the entire vector: all elements must be the same type. When you attempt to combine different types they will be coerced in a fixed order: character → double → integer → logical. For example, combining a character and an integer yields a character:

str(c("a", 1))
#>  chr [1:2] "a" "1"

Coercion often happens automatically. Most mathematical functions (+, log, abs, etc.) will coerce to numeric. This coercion is particularly useful for logical vectors because TRUE becomes 1 and FALSE becomes 0.

x <- c(FALSE, FALSE, TRUE)
as.numeric(x)
#> [1] 0 0 1

# Total number of TRUEs
sum(x)
#> [1] 1

# Proportion that are TRUE
mean(x)
#> [1] 0.333

Generally, you can deliberately coerce by using an as.*() function, like as.logical(), as.integer(), as.double(), or as.character(). Failed coercion of strings generates a warning and a missing value:

as.integer(c("1", "1.5", "a"))
#> Warning: NAs introduced by coercion
#> [1]  1  1 NA

1. How do you create raw and complex scalars? (See ?raw and ?complex.)

2. Test your knowledge of the vector coercion rules by predicting the output of the following uses of c():

   c(1, FALSE)
   c("a", 1)
   c(TRUE, 1L)
   

3. Why is 1 == ”1” true? Why is -1 < FALSE true? Why is ”one” < 2 false?

4. Why is the default missing value, NA, a logical vector? What’s special about logical vectors? (Hint: think about c(FALSE, NA_character_).)

5. Precisely what do is.atomic(), is.numeric(), and is.vector() test for?

You can think of attributes as name-value pairs⁷ that attach metadata to an object. Individual attributes can be retrieved and modified with attr(), or retrieved en masse with attributes(), and set en masse with structure().

a <- 1:3
attr(a, "x") <- "abcdef"
attr(a, "x")
#> [1] "abcdef"

attr(a, "y") <- 4:6
str(attributes(a))
#> List of 2
#> $ x: chr "abcdef"
#> $ y: int [1:3] 4 5 6

# Or equivalently
a <- structure(
  1:3,
  x = "abcdef",
  y = 4:6
)
str(attributes(a))
#> List of 2
#>  $ x: chr "abcdef"
#>  $ y: int [1:3] 4 5 6

Attributes should generally be thought of as ephemeral. For example, most attributes are lost by most operations:

attributes(a[1])
#> NULL
attributes(sum(a))
#> NULL

There are only two attributes that are routinely preserved:

• names, a character vector giving each element a name.

• dim, short for dimensions, an integer vector, used to turn vectors into matrices or arrays.

To preserve other attributes, you’ll need to create your own S3 class, the topic of Chapter 13.

You can name a vector in three ways:

# When creating it:
x <- c(a = 1, b = 2, c = 3)

# By assigning a character vector to names()
x <- 1:3
names(x) <- c("a", "b", "c")

# Inline, with setNames():
x <- setNames(1:3, c("a", "b", "c"))

Avoid using attr(x, ”names”) as it requires more typing and is less readable than names(x). You can remove names from a vector by using unname(x) or names(x) <- NULL.

To be technically correct, when drawing the named vector x, I should draw it like so:

However, names are so special and so important, that unless I’m trying specifically to draw attention to the attributes data structure, I’ll use them to label the vector directly:

To be useful with character subsetting (e.g. Section 4.5.1) names should be unique, and non-missing, but this is not enforced by R. Depending on how the names are set, missing names may be either ”” or NA_character_. If all names are missing, names() will return NULL.

Adding a dim attribute to a vector allows it to behave like a 2-dimensional matrix or a multi-dimensional array. Matrices and arrays are primarily mathematical and statistical tools, not programming tools, so they’ll be used infrequently and only covered briefly in this book. Their most important feature is multidimensional subsetting, which is covered in Section 4.2.3.

You can create matrices and arrays with matrix() and array(), or by using the assignment form of dim():

# Two scalar arguments specify row and column sizes
a <- matrix(1:6, nrow = 2, ncol = 3)
a
#>      [,1] [,2] [,3]
#> [1,]    1    3    5
#> [2,]    2    4    6

# One vector argument to describe all dimensions
b <- array(1:12, c(2, 3, 2))
b
#> , , 1
#>
#>      [,1] [,2] [,3]
#> [1,]    1    3    5
#> [2,]    2    4    6
#>
#> , , 2
#>
#>      [,1] [,2] [,3]
#> [1,]    7    9 11
#> [2,]    8 10 12

# You can also modify an object in place by setting dim()
c <- 1:6
dim(c) <- c(3, 2)
c
#>      [,1] [,2]
#> [1,]    1    4
#> [2,]    2    5
#> [3,]    3    6

Many of the functions for working with vectors have generalisations for matrices and arrays:

A vector without a dim attribute set is often thought of as 1-dimensional, but actually has NULL dimensions. You also can have matrices with a single row or single column, or arrays with a single dimension. They may print similarly, but will behave differently. The differences aren’t too important, but it’s useful to know they exist in case you get strange output from a function (tapply() is a frequent offender). As always, use str() to reveal the differences.

str(1:3)                       # 1d vector
#>  int [1:3] 1 2 3
str(matrix(1:3, ncol = 1))     # column vector
#>  int  [1:3, 1] 1 2 3
str(matrix(1:3, nrow = 1))     # row vector
#>  int  [1, 1:3] 1 2 3
str(array(1:3, 3))             # "array" vector
#>  int [1:3(1d)] 1 2 3

1. How is setNames() implemented? How is unname() implemented? Read the source code.

2. What does dim() return when applied to a 1-dimensional vector? When might you use NROW() or NCOL()?

3. How would you describe the following three objects? What makes them different from 1:5?

   x1 <- array(1:5, c(1, 1, 5))
   x2 <- array(1:5, c(1, 5, 1))
   x3 <- array(1:5, c(5, 1, 1))
   

4. An early draft used this code to illustrate structure():

   structure(1:5, comment = "my attribute")
   #> [1] 1 2 3 4 5
   

   But when you print that object you don’t see the comment attribute. Why? Is the attribute missing, or is there something else special about it? (Hint: try using help.)

A factor is a vector that can contain only predefined values. It is used to store categorical data. Factors are built on top of an integer vector with two attributes: a class, “factor”, which makes it behave differently from regular integer vectors, and levels, which defines the set of allowed values.

x <- factor(c("a", "b", "b", "a"))
x
#> [1] a b b a
#> Levels: a b

typeof(x)
#> [1] "integer"
attributes(x)
#> $levels
#> [1] "a" "b"
#>
#> $class
#> [1] "factor"

Factors are useful when you know the set of possible values but they’re not all present in a given dataset. In contrast to a character vector, when you tabulate a factor you’ll get counts of all categories, even unobserved ones:

sex_char <- c("m", "m", "m")
sex_factor <- factor(sex_char, levels = c("m", "f"))

table(sex_char)
#> sex_char
#> m
#> 3
table(sex_factor)
#> sex_factor
#> m f
#> 3 0

Ordered factors are a minor variation of factors. In general, they behave like regular factors, but the order of the levels is meaningful (low, medium, high) (a property that is automatically leveraged by some modelling and visualisation functions).

grade <- ordered(c("b", "b", "a", "c"), levels = c("c", "b", "a"))
grade
#> [1] b b a c
#> Levels: c < b < a

In base R⁸ you tend to encounter factors very frequently because many base R functions (like read.csv() and data.frame()) automatically convert character vectors to factors. This is suboptimal because there’s no way for those functions to know the set of all possible levels or their correct order: the levels are a property of theory or experimental design, not of the data. Instead, use the argument stringsAsFactors = FALSE to suppress this behaviour, and then manually convert character vectors to factors using your knowledge of the “theoretical” data. To learn about the historical context of this behaviour, I recommend stringsAsFactors: An unauthorized biography (http://simplystatistics.org/2015/07/24/stringsasfactors-an-unauthorized-biography/) by Roger Peng, and stringsAsFactors = <sigh> (http://notstatschat.tumblr.com/post/124987394001/stringsasfactors-sigh) by Thomas Lumley.

While factors look like (and often behave like) character vectors, they are built on top of integers. So be careful when treating them like strings. Some string methods (like gsub() and grepl()) will automatically coerce factors to strings, others (like nchar()) will throw an error, and still others will (like c()) use the underlying integer values. For this reason, it’s usually best to explicitly convert factors to character vectors if you need string-like behaviour.

Date vectors are built on top of double vectors. They have class “Date” and no other attributes:

today <- Sys.Date()

typeof(today)
#> [1] "double"
attributes(today)

#> $class
#> [1] "Date"

The value of the double (which can be seen by stripping the class), represents the number of days since 1970-01-01⁹:

date <- as.Date("1970-02-01")
unclass(date)
#> [1] 31

Base R¹⁰ provides two ways of storing date-time information, POSIXct, and POSIXlt. These are admittedly odd names: “POSIX” is short for Portable Operating System Interface, which is a family of cross-platform standards. “ct” standards for calendar time (the time_t type in C), and “lt” for local time (the struct tm type in C). Here we’ll focus on POSIXct, because it’s the simplest, is built on top of an atomic vector, and is most appropriate for use in data frames. POSIXct vectors are built on top of double vectors, where the value represents the number of seconds since 1970-01-01.

now_ct <- as.POSIXct("2018-08-01 22:00", tz = "UTC")
now_ct
#> [1] "2018-08-01 22:00:00 UTC"

typeof(now_ct)
#> [1] "double"
attributes(now_ct)
#> $class
#> [1] "POSIXct" "POSIXt"
#>
#> $tzone
#> [1] "UTC"

The tzone attribute controls only how the date-time is formatted; it does not control the instant of time represented by the vector. Note that the time is not printed if it is midnight.

structure(now_ct, tzone = "Asia/Tokyo")
#> [1] "2018-08-02 07:00:00 JST"
structure(now_ct, tzone = "America/New_York")
#> [1] "2018-08-01 18:00:00 EDT"
structure(now_ct, tzone = "Australia/Lord_Howe")
#> [1] "2018-08-02 08:30:00 +1030"
structure(now_ct, tzone = "Europe/Paris")
#> [1] "2018-08-02 CEST"

Durations, which represent the amount of time between pairs of dates or date-times, are stored in difftimes. Difftimes are built on top of doubles, and have a units attribute that determines how the integer should be interpreted:

one_week_1 <- as.difftime(1, units = "weeks")
one_week_1
#> Time difference of 1 weeks

typeof(one_week_1)
#> [1] "double"
attributes(one_week_1)
#> $class
#> [1] "difftime"
#>
#> $units
#> [1] "weeks"

one_week_2 <- as.difftime(7, units = "days")
one_week_2
#> Time difference of 7 days

typeof(one_week_2)
#> [1] "double"
attributes(one_week_2)
#> $class
#> [1] "difftime"
#>
#> $units
#> [1] "days"

1. What sort of object does table() return? What is its type? What attributes does it have? How does the dimensionality change as you tabulate more variables?

2. What happens to a factor when you modify its levels?

   f1 <- factor(letters)
   levels(f1) <- rev(levels(f1))
   

3. What does this code do? How do f2 and f3 differ from f1?

   f2 <- rev(factor(letters))
   
   f3 <- factor(letters, levels = rev(letters))
   

You construct lists with list():

l1 <- list(
  1:3,
  "a",
  c(TRUE, FALSE, TRUE),
  c(2.3, 5.9)
)

typeof(l1)
#> [1] "list"

str(l1)
#> List of 4
#> $ :  int [1:3] 1 2 3
#> $ :  chr "a"
#> $ :  logi [1:3] TRUE FALSE TRUE
#> $ :  num [1:2] 2.3 5.9

Because the elements of a list are references, creating a list does not involve copying the components into the list. For this reason, the total size of a list might be smaller than you might expect.

lobstr::obj_size(mtcars)
#> 7,208 B

l2 <- list(mtcars, mtcars, mtcars, mtcars)
lobstr::obj_size(l2)
#> 7,288 B

Lists can contain complex objects so it’s not possible to pick a single visual style that works for every list. Generally I’ll draw lists like vectors, using colour to remind you of the hierarchy.

Lists are sometimes called recursive vectors because a list can contain other lists. This makes them fundamentally different from atomic vectors.

l3 <- list(list(list(1)))
str(l3)
#> List of 1
#>  $ :List of 1
#>   ..$ :List of 1
#>   .. ..$ : num 1

c() will combine several lists into one. If given a combination of atomic vectors and lists, c() will coerce the vectors to lists before combining them. Compare the results of list() and c():

l4 <- list(list(1, 2), c(3, 4))
l5 <- c(list(1, 2), c(3, 4))
str(l4)
#> List of 2
#>  $ :List of 2
#>   ..$ : num 1
#>   ..$ : num 2
#>  $ : num [1:2] 3 4
str(l5)
#> List of 4
#> $ : num 1
#> $ : num 2
#> $ : num 3
#> $ : num 4

The typeof() a list is list. You can test for a list with is.list(), and coerce to a list with as.list().

list(1:3)
#> [[1]]
#> [1] 1 2 3
as.list(1:3)
#> [[1]]
#> [1] 1
#>
#> [[2]]
#> [1] 2
#>
#> [[3]]
#> [1] 3

You can turn a list into an atomic vector with unlist(). The rules for the resulting type are complex, not well documented, and not always equivalent to what you’d get with c().

With atomic vectors, the dimension attribute is commonly used to create matrices. With lists, the dimension attribute can be used to create list-matrices or list-arrays:

l <- list(1:3, "a", TRUE, 1.0)
dim(l) <- c(2, 2)
l
#>      [,1]      [,2]
#> [1,] Integer,3 TRUE
#> [2,] "a"       1

l[[1, 1]]
#> [1] 1 2 3

These data structures are relatively esoteric but they can be useful if you want to arrange objects in a grid-like structure. For example, if you’re running models on a spatio-temporal grid, it might be more intuitive to store the models in a 3D array that matches the grid structure.

1. List all the ways that a list differs from an atomic vector.

2. Why do you need to use unlist() to convert a list to an atomic vector? Why doesn’t as.vector() work?

3. Compare and contrast c() and unlist() when combining a date and date-time into a single vector.

You create a data frame by supplying name-vector pairs to data.frame():

df <- data.frame(
  x = 1:3,
  y = c("a", "b", "c")
)
str(df)
#> 'data.frame':    3 obs. of  2 variables:

#> $ x: int 1 2 3
#> $ y: Factor w/ 3 levels "a","b","c": 1 2 3

Beware of the default conversion of strings to factors. Use stringsAsFactors = FALSE to suppress this and keep character vectors as character vectors:

df1 <- data.frame(
  x = 1:3,
  y = c("a", "b", "c"),
  stringsAsFactors = FALSE
)
str(df1)
#> 'data.frame':    3 obs. of     2 variables:
#> $ x: int 1 2 3
#> $ y: chr "a" "b" "c"

Creating a tibble is similar to creating a data frame. The difference between the two is that tibbles never coerce their input (this is one feature that makes them lazy):

df2 <- tibble(
  x = 1:3,
  y = c("a", "b", "c")
)
str(df2)
#> Classes 'tbl_df', 'tbl' and 'data.frame':         3 obs. of    2 variables:
#> $ x: int 1 2 3
#> $ y: chr "a" "b" "c"

Additionally, while data frames automatically transform non-syntactic names (unless check.names = FALSE), tibbles do not (although they do print non-syntactic names surrounded by `).

names(data.frame(`1` = 1))
#> [1] "X1"

names(tibble(`1` = 1))
#> [1] "1"

While every element of a data frame (or tibble) must have the same length, both data.frame() and tibble() will recycle shorter inputs. However, while data frames automatically recycle columns that are an integer multiple of the longest column, tibbles will only recycle vectors of length one.

data.frame(x = 1:4, y = 1:2)
#>   x y
#> 1 1 1
#> 2 2 2
#> 3 3 1
#> 4 4 2
data.frame(x = 1:4, y = 1:3)
#> Error in data.frame(x = 1:4, y = 1:3): arguments imply differing
#> number of rows: 4, 3

tibble(x = 1:4, y = 1)
#> # A tibble: 4 x 2
#>       x     y
#>   <int> <dbl>
#> 1     1     1
#> 2     2     1
#> 3     3     1
#> 4     4     1
tibble(x = 1:4, y = 1:2)
#> Error: Tibble columns must have consistent lengths, only values of
#> length one are recycled:
#> * Length 2: Column `y`
#> * Length 4: Column `x`

There is one final difference: tibble() allows you to refer to variables created during construction:

tibble(
  x = 1:3,
  y = x * 2
)
#> # A tibble: 3 x 2
#>       x     y
#>   <int> <dbl>
#> 1     1     2
#> 2     2     4
#> 3     3     6

(Inputs are evaluated left-to-right.)

When drawing data frames and tibbles, rather than focussing on the implementation details, i.e. the attributes:

I’ll draw them the same way as a named list, but arrange them to emphasise their columnar structure.

Data frames allow you to label each row with a name, a character vector containing only unique values:

df3 <- data.frame(
  age = c(35, 27, 18),
  hair = c("blond", "brown", "black"),
  row.names = c("Bob", "Susan", "Sam")
)
df3
#>       age hair
#> Bob   35 blond
#> Susan 27 brown
#> Sam   18 black

You can get and set row names with rownames(), and you can use them to subset rows:

rownames(df3)
#> [1] "Bob" "Susan" "Sam"

df3["Bob", ]
#>     age hair
#> Bob 35 blond

Row names arise naturally if you think of data frames as 2D structures like matrices: columns (variables) have names so rows (observations) should too. Most matrices are numeric, so having a place to store character labels is important. But this analogy to matrices is misleading because matrices possess an important property that data frames do not: they are transposable. In matrices the rows and columns are interchangeable, and transposing a matrix gives you another matrix (transposing again gives you the original matrix). With data frames, however, the rows and columns are not interchangeable: the transpose of a data frame is not a data frame.

There are three reasons why row names are undesirable:

• Metadata is data, so storing it in a different way to the rest of the data is fundamentally a bad idea. It also means that you need to learn a new set of tools to work with row names; you can’t use what you already know about manipulating columns.

• Row names are a poor abstraction for labelling rows because they only work when a row can be identified by a single string. This fails in many cases, for example when you want to identify a row by a non-character vector (e.g. a time point), or with multiple vectors (e.g. position, encoded by latitude and longitude).

• Row names must be unique, so any duplication of rows (e.g. from bootstrapping) will create new row names. If you want to match rows from before and after the transformation, you’ll need to perform complicated string surgery.

  df3[c(1,  1, 1), ]
  #>        age  hair
  #> Bob    35  blond
  #> Bob.1  35  blond
  #> Bob.2  35  blond
  

For these reasons, tibbles do not support row names. Instead the tibble package provides tools to easily convert row names into a regular column with either rownames_to_column(), or the rownames argument in as_tibble():

as_tibble(df3, rownames = "name")
#> # A tibble: 3 x 3
#>   name    age hair
#>   <chr> <dbl> <fct>

#> 1 Bob     35 blond
#> 2 Susan   27 brown
#> 3 Sam     18 black

One of the most obvious differences between tibbles and data frames is how they print. I assume that you’re already familiar with how data frames are printed, so here I’ll highlight some of the biggest differences using an example dataset included in the dplyr package:

dplyr::starwars
#> # A tibble: 87 x 13
#>    name height mass hair_color skin_color eye_color birth_year
#>    <chr> <int> <dbl> <chr>     <chr>      <chr>          <dbl>
#> 1 Luke      172    77 blond     fair       blue            19
#> 2 C-3PO    167    75 <NA>      gold       yellow         112
#> 3 R2-D2     96    32 <NA>      white, bl&hellip; red       33
#> 4 Dart&hellip;      202  136 none       white    yellow            41.9
#> 5 Leia&hellip;      150   49 brown      light    brown             19
#> 6 Owen&hellip;      178  120 brown, gr light          blue             52
#> 7 Beru&hellip;      165   75 brown      light      blue             47
#> 8 R5-D4      97  32  <NA>      white, red red              NA
#> 9 Bigg&hellip;      183   84 black      light      brown            24
#> 10 Obi-&hellip;     182   77 auburn, w fair         blue-gray        57
#> #&hellip;    with 77 more rows, and 6 more variables: gender <chr>,
#> #   homeworld <chr>, species <chr>, films <list>, vehicles <list>,
#> #   starships <list>

• Tibbles only show the first 10 rows and all the columns that will fit on screen. Additional columns are shown at the bottom.

• Each column is labelled with its type, abbreviated to three or four letters.

• Wide columns are truncated to avoid having a single long string occupy an entire row. (This is still a work in progress: it’s a tricky tradeoff between showing as many columns as possible and showing columns in their entirety.)

• When used in console environments that support it, colour is used judiciously to highlight important information, and de-emphasise supplemental details.

As you will learn in Chapter 4, you can subset a data frame or a tibble like a 1D structure (where it behaves like a list), or a 2D structure (where it behaves like a matrix).

In my opinion, data frames have two undesirable subsetting behaviours:

• When you subset columns with df[, vars], you will get a vector if vars selects one variable, otherwise you’ll get a data frame. This is a frequent source of bugs when using [ in a function, unless you always remember to use df[, vars, drop = FALSE].

• When you attempt to extract a single column with df$x and there is no column x, a data frame will instead select any variable that starts with x. If no variable starts with x, df$x will return NULL. This makes it easy to select the wrong variable or to select a variable that doesn’t exist.

Tibbles tweak these behaviours so that a [ always returns a tibble, and a $ doesn’t do partial matching and warns if it can’t find a variable (this is what makes tibbles surly).

df1 <- data.frame(xyz = "a")
df2 <- tibble(xyz = "a")

str(df1$x)
#> Factor w/ 1 level "a": 1
str(df2$x)
#> Warning: Unknown or uninitialised column: 'x'.
#> NULL

A tibble’s insistence on returning a data frame from [ can cause problems with legacy code, which often uses df[, ”col”] to extract a single column. If you want a single column, I recommend using df[[”col”]]. This clearly communicates your intent, and works with both data frames and tibbles.

To check if an object is a data frame or tibble, use is.data.frame():

is.data.frame(df1)
#> [1] TRUE
is.data.frame(df2)
#> [1] TRUE

Typically, it should not matter if you have a tibble or data frame, but if you need to be certain, use is_tibble():

is_tibble(df1)
#> [1] FALSE
is_tibble(df2)
#> [1] TRUE

You can coerce an object to a data frame with as.data.frame() or to a tibble with as_tibble().

Since a data frame is a list of vectors, it is possible for a data frame to have a column that is a list. This is very useful because a list can contain any other object: this means you can put any object in a data frame. This allows you to keep related objects together in a row, no matter how complex the individual objects are. You can see an application of this in the “Many Models” chapter of R for Data Science, http://r4ds.had.co.nz/many-models.html.

List-columns are allowed in data frames but you have to do a little extra work by either adding the list-column after creation or wrapping the list in I()¹³.

df <- data.frame(x = 1:3)
df$y <- list(1:2, 1:3, 1:4)

data.frame(
  x = 1:3,
  y = I(list(1:2, 1:3, 1:4))
)
#>   x          y
#> 1 1       1, 2
#> 2 2    1, 2, 3
#> 3 3 1, 2, 3, 4

List columns are easier to use with tibbles because they can be directly included inside tibble() and they will be printed tidily:

tibble(
  x = 1:3,
  y = list(1:2, 1:3, 1:4)
)
#> # A tibble: 3 x 2
#>       x y
#> <int> <list>
#> 1   1 <int [2]>
#> 2   2 <int [3]>
#> 3   3 <int [4]>

As long as the number of rows matches the data frame, it’s also possible to have a matrix or array as a column of a data frame. (This requires a slight extension to our definition of a data frame: it’s not the length() of each column that must be equal, but the NROW().) As for list-columns, you must either add it after creation, or wrap it in I().

dfm <- data.frame(
  x = 1:3 * 10
)
dfm$y <- matrix(1:9, nrow = 3)
dfm$z <- data.frame(a = 3:1, b = letters[1:3], stringsAsFactors = FALSE)

str(dfm)
#> 'data.frame':  3 obs. of 3 variables:
#> $ x: num 10 20 30
#> $ y: int [1:3, 1:3] 1 2 3 4 5 6 7 8 9
#> $ z:'data.frame': 3 obs. of 2 variables:

#> ..$ a: int 3 2 1
#> ..$ b: chr "a" "b" "c"

Matrix and data frame columns require a little caution. Many functions that work with data frames assume that all columns are vectors. Also, the printed display can be confusing.

dfm[1, ]
#>    x y.1 y.2 y.3 z.a z.b
#> 1 10   1   4   7   3   a

1. Can you have a data frame with zero rows? What about zero columns?

2. What happens if you attempt to set rownames that are not unique?

3. If df is a data frame, what can you say about t(df), and t(t(df))? Perform some experiments, making sure to try different column types.

4. What does as.matrix() do when applied to a data frame with columns of different types? How does it differ from data.matrix()?

Take this short quiz to determine if you need to read this chapter. If the answers quickly come to mind, you can comfortably skip this chapter. Check your answers in Section 4.6.

1. What is the result of subsetting a vector with positive integers, negative integers, a logical vector, or a character vector?

2. What’s the difference between [, [[, and $ when applied to a list?

3. When should you use drop = FALSE?

4. If x is a matrix, what does x[] <- 0 do? How is it different from x <- 0?

5. How can you use a named vector to relabel categorical variables?

• Section 4.2 starts by teaching you about [. You’ll learn the six ways to subset atomic vectors. You’ll then learn how those six ways act when used to subset lists, matrices, and data frames.

• Section 4.3 expands your knowledge of subsetting operators to include [[ and $ and focuses on the important principles of simplifying versus preserving.

• In Section 4.4 you’ll learn the art of subassignment, which combines subsetting and assignment to modify parts of an object.

• Section 4.5 leads you through eight important, but not obvious, applications of subsetting to solve problems that you often encounter in data analysis.

Let’s explore the different types of subsetting with a simple vector, x.

x <- c(2.1, 4.2, 3.3, 5.4)

Note that the number after the decimal point represents the original position in the vector.

There are six things that you can use to subset a vector:

• Positive integers return elements at the specified positions:

  x[c(3, 1)]
  #> [1] 3.3 2.1
  x[order(x)]
  #> [1] 2.1 3.3 4.2 5.4
  
  # Duplicate indices will duplicate values
  x[c(1, 1)]
  #> [1] 2.1 2.1
  

  # Real numbers are silently truncated to integers
  x[c(2.1, 2.9)]
  #> [1] 4.2 4.2
  

• Negative integers exclude elements at the specified positions:

  x[-c(3, 1)]
  #> [1] 4.2 5.4
  

  Note that you can’t mix positive and negative integers in a single subset:

  x[c(-1, 2)]
  #> Error in x[c(-1, 2)]: only 0's may be mixed with negative subscripts
  

• Logical vectors select elements where the corresponding logical value is TRUE. This is probably the most useful type of subsetting because you can write an expression that uses a logical vector:

  x[c(TRUE, TRUE, FALSE, FALSE)]
  #> [1] 2.1 4.2
  x[x > 3]
  #> [1] 4.2 3.3 5.4
  

  In x[y], what happens if x and y are different lengths? The behaviour is controlled by the recycling rules where the shorter of the two is recycled to the length of the longer. This is convenient and easy to understand when one of x and y is length one, but I recommend avoiding recycling for other lengths because the rules are inconsistently applied throughout base R.

  x[c(TRUE, FALSE)]
  #> [1] 2.1 3.3
  # Equivalent to
  x[c(TRUE, FALSE, TRUE, FALSE)]
  #> [1] 2.1 3.3
  

  Note that a missing value in the index always yields a missing value in the output:

  x[c(TRUE, TRUE, NA, FALSE)]
  #> [1] 2.1 4.2 NA
  

• Nothing returns the original vector. This is not useful for 1D vectors, but, as you’ll see shortly, is very useful for matrices, data frames, and arrays. It can also be useful in conjunction with assignment.

  x[]
  #> [1] 2.1 4.2 3.3 5.4
  

• Zero returns a zero-length vector. This is not something you usually do on purpose, but it can be helpful for generating test data.

  x[0]
  #> numeric(0)
  

• If the vector is named, you can also use character vectors to return elements with matching names.

  (y <- setNames(x, letters[1:4]))
  #>   a   b   c   d
  #> 2.1 4.2 3.3 5.4
  y[c("d", "c", "a")]
  #>   d   c   a
  #> 5.4 3.3 2.1
  
  # Like integer indices, you can repeat indices
  y[c("a", "a", "a")]
  #>   a   a   a
  #> 2.1 2.1 2.1
  
  # When subsetting with [, names are always matched exactly
  z <- c(abc = 1, def = 2)
  z[c("a", "d")]
  #> <NA> <NA>
  #>   NA   NA
  

  NB: Factors are not treated specially when subsetting. This means that subsetting will use the underlying integer vector, not the character levels. This is typically unexpected, so you should avoid subsetting with factors:

  y[factor("b")]
  #> a
  #> 2.1
  

Subsetting a list works in the same way as subsetting an atomic vector. Using [ always returns a list; [[ and $, as described in Section 4.3, lets you pull out elements of a list.

You can subset higher-dimensional structures in three ways:

• With multiple vectors.

• With a single vector.

• With a matrix.

The most common way of subsetting matrices (2D) and arrays (>2D) is a simple generalisation of 1D subsetting: supply a 1D index for each dimension, separated by a comma. Blank subsetting is now useful because it lets you keep all rows or all columns.

a <- matrix(1:9, nrow = 3)
colnames(a) <- c("A", "B", "C")
a[1:2, ]
#>      A B C
#> [1,] 1 4 7
#> [2,] 2 5 8
a[c(TRUE, FALSE, TRUE), c("B", "A")]
#>      B A
#> [1,] 4 1
#> [2,] 6 3
a[0, -2]
#>      A C

By default, [ simplifies the results to the lowest possible dimensionality. For example, both of the following expressions return 1D vectors. You’ll learn how to avoid “dropping” dimensions in Section 4.2.5:

a[1,   ]
#> A B C
#> 1 4 7
a[1, 1]
#> A
#> 1

Because both matrices and arrays are just vectors with special attributes, you can subset them with a single vector, as if they were a 1D vector. Note that arrays in R are stored in column-major order:

vals <- outer(1:5, 1:5, FUN = "paste", sep = ",")
vals
#>      [,1]  [,2]  [,3]  [,4]  [,5]
#> [1,] "1,1" "1,2" "1,3" "1,4" "1,5"
#> [2,] "2,1" "2,2" "2,3" "2,4" "2,5"
#> [3,] "3,1" "3,2" "3,3" "3,4" "3,5"
#> [4,] "4,1" "4,2" "4,3" "4,4" "4,5"
#> [5,] "5,1" "5,2" "5,3" "5,4" "5,5"

vals[c(4, 15)]
#> [1] "4,1" "5,3"

You can also subset higher-dimensional data structures with an integer matrix (or, if named, a character matrix). Each row in the matrix specifies the location of one value, and each column corresponds to a dimension in the array. This means that you can use a 2 column matrix to subset a matrix, a 3 column matrix to subset a 3D array, and so on. The result is a vector of values:

select <- matrix(ncol = 2, byrow = TRUE, c(
  1, 1,
  3, 1,
  2, 4
))
vals[select]
#> [1] "1,1" "3,1" "2,4"

Data frames have the characteristics of both lists and matrices:

• When subsetting with a single index, they behave like lists and index the columns, so df[1:2] selects the first two columns.

• When subsetting with two indices, they behave like matrices, so df[1:3, ] selects the first three rows (and all the columns)¹.

df <- data.frame(x = 1:3, y = 3:1, z = letters[1:3])

df[df$x == 2, ]
#>   x y z
#> 2 2 2 b
df[c(1, 3), ]
#>   x y z
#> 1 1 3 a
#> 3 3 1 c

# There are two ways to select columns from a data frame
# Like a list
df[c("x", "z")]
#>   x z
#> 1 1 a
#> 2 2 b
#> 3 3 c
# Like a matrix
df[, c("x", "z")]
#>   x z
#> 1 1 a
#> 2 2 b
#> 3 3 c

# There's an important difference if you select a single
# column: matrix subsetting simplifies by default, list
# subsetting does not.
str(df["x"])
#> 'data.frame':   3 obs. of 1 variable:
#> $ x: int   1 2 3
str(df[, "x"])
#> int [1:3]  1 2 3

Subsetting a tibble with [ always returns a tibble:

df <- tibble::tibble(x = 1:3, y = 3:1, z = letters[1:3])

str(df["x"])
#> Classes 'tbl_df', 'tbl' and 'data.frame':   3 obs. of   1 variable:
#>  $ x: int  1 2 3
str(df[, "x"])
#> Classes 'tbl_df', 'tbl' and 'data.frame':   3 obs. of   1 variable:
#>  $ x: int  1 2 3

By default, subsetting a matrix or data frame with a single number, a single name, or a logical vector containing a single TRUE, will simplify the returned output, i.e. it will return an object with lower dimensionality. To preserve the original dimensionality, you must use drop = FALSE.

• For matrices and arrays, any dimensions with length 1 will be dropped:

  a <- matrix(1:4, nrow = 2)
  str(a[1, ])
  #>  int [1:2] 1 3
  
  str(a[1, , drop = FALSE])
  #>  int [1, 1:2] 1 3
  

• Data frames with a single column will return just that column:

  df <- data.frame(a = 1:2, b = 1:2)
  str(df[, "a"])
  #> int [1:2] 1 2
  
  str(df[, "a", drop = FALSE])
  #> 'data.frame':    2 obs. of  1 variable:
  #> $ a: int 1 2
  

The default drop = TRUE behaviour is a common source of bugs in functions: you check your code with a data frame or matrix with multiple columns, and it works. Six months later, you (or someone else) uses it with a single column data frame and it fails with a mystifying error. When writing functions, get in the habit of always using drop = FALSE when subsetting a 2D object. For this reason, tibbles default to drop = FALSE, and [ always returns another tibble.

Factor subsetting also has a drop argument, but its meaning is rather different. It controls whether or not levels (rather than dimensions) are preserved, and it defaults to FALSE. If you find you’re using drop = TRUE a lot it’s often a sign that you should be using a character vector instead of a factor.

z <- factor(c("a", "b"))
z[1]
#> [1] a
#> Levels: a b
z[1, drop = TRUE]
#> [1] a
#> Levels: a

1. Fix each of the following common data frame subsetting errors:

   mtcars[mtcars$cyl = 4, ]
   mtcars[-1:4, ]
   mtcars[mtcars$cyl <= 5]
   mtcars[mtcars$cyl == 4 | 6, ]
   

2. Why does the following code yield five missing values? (Hint: why is it different from x[NA_real_]?)

   x <- 1:5
   x[NA]
   #> [1] NA NA NA NA NA
   

3. What does upper.tri() return? How does subsetting a matrix with it work? Do we need any additional subsetting rules to describe its behaviour?

   x <- outer(1:5, 1:5, FUN = "*")
   x[upper.tri(x)]
   

4. Why does mtcars[1:20] return an error? How does it differ from the similar mtcars[1:20, ]?

5. Implement your own function that extracts the diagonal entries from a matrix (it should behave like diag(x) where x is a matrix).

6. What does df[is.na(df)] <- 0 do? How does it work?

[[ is most important when working with lists because subsetting a list with [ always returns a smaller list. To help make this easier to understand we can use a metaphor:

If list x is a train carrying objects, then x[[5]] is the object in car 5; x[4:6] is a train of cars 4-6.

Let’s use this metaphor to make a simple list:

x <- list(1:3, "a", 4:6)

When extracting a single element, you have two options: you can create a smaller train, i.e., fewer carriages, or you can extract the contents of a particular carriage. This is the difference between [ and [[:

When extracting multiple (or even zero!) elements, you have to make a smaller train:

Because [[ can return only a single item, you must use it with either a single positive integer or a single string. If you use a vector with [[, it will subset recursively, i.e. x[[c(1, 2)]] is equivalent to x[[1]][[2]]. This is a quirky feature that few know about, so I recommend avoiding it in favour of purrr::pluck(), which you’ll learn about in Section 4.3.3.

While you must use [[ when working with lists, I’d also recommend using it with atomic vectors whenever you want to extract a single value. For example, instead of writing:

for (i in 2:length(x)) {
  out[i] <- fun(x[i], out[i - 1])
}

It’s better to write:

for (i in 2:length(x)) {
  out[[i]] <- fun(x[[i]], out[[i - 1]])
}

Doing so reinforces the expectation that you are getting and setting individual values.

$ is a shorthand operator: x$y is roughly equivalent to x[[”y”]]. It’s often used to access variables in a data frame, as in mtcars$cyl or diamonds$carat. One common mistake with $ is to use it when you have the name of a column stored in a variable:

var <- "cyl"
# Doesn't work - mtcars$var translated to mtcars[["var"]]
mtcars$var
#> NULL

# Instead use [[
mtcars[[var]]
#> [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4

The one important difference between $ and [[ is that $ does (left-to-right) partial matching:

x <- list(abc = 1)
x$a
#> [1] 1
x[["a"]]
#> NULL

To help avoid this behaviour I highly recommend setting the global option warnPartialMatchDollar to TRUE:

options(warnPartialMatchDollar = TRUE)
x$a
#> Warning in x$a: partial match of 'a' to 'abc'
#> [1] 1

(For data frames, you can also avoid this problem by using tibbles, which never do partial matching.)

It’s useful to understand what happens with [[ when you use an “invalid” index. The following table summarise what happens when you subset a logical vector, list, and NULL with a zero-length object (like NULL or logical()), out-of-bounds values (OOB), or a missing value (e.g. NA_integer_) with [[. Each cell shows the result of subsetting the data structure named in the row by the type of index described in the column. I’ve only shown the results for logical vectors, but other atomic vectors behave similarly, returning elements of the same type (NB: int = integer; chr = character).

If the vector being indexed is named, then the names of OOB, missing, or NULL components will be <NA>.

The inconsistencies in the table above led to the development of purrr::pluck() and purrr::chuck(). When the element is missing, pluck() always returns NULL (or the value of the .default argument) and chuck() always throws an error. The behaviour of pluck() makes it well suited for indexing into deeply nested data structures where the component you want may not exist (as is common when working with JSON data from web APIs). pluck() also allows you to mix integer and character indices, and provides an alternative default value if an item does not exist:

x <- list(
  a = list(1, 2, 3),
  b = list(3, 4, 5)
)

purrr::pluck(x, "a", 1)
#> [1] 1

purrr::pluck(x, "c", 1)
#> NULL

purrr::pluck(x, "c", 1, .default = NA)
#> [1] NA

There are two additional subsetting operators, which are needed for S4 objects: @ (equivalent to $), and slot() (equivalent to [[). @ is more restrictive than $ in that it will return an error if the slot does not exist. These are described in more detail in Chapter 15.

1. Brainstorm as many ways as possible to extract the third value from the cyl variable in the mtcars dataset.

2. Given a linear model, e.g., mod <- lm(mpg ~ wt, data = mtcars), extract the residual degrees of freedom. Then extract the R squared from the model summary (summary(mod))

Character matching is a powerful way to create lookup tables. Say you want to convert abbreviations:

x <- c("m", "f", "u", "f", "f", "m", "m")
lookup <- c(m = "Male", f = "Female", u = NA)
lookup[x]
#>        m        f        u        f        f        m         m
#>   "Male" "Female"       NA "Female" "Female"   "Male"    "Male"

Note that if you don’t want names in the result, use unname() to remove them.

unname(lookup[x])
#> [1] "Male"  "Female" NA     "Female" "Female" "Male"    "Male"

You can also have more complicated lookup tables with multiple columns of information. For example, suppose we have a vector of integer grades, and a table that describes their properties:

grades <- c(1, 2, 2, 3, 1)

info <- data.frame(
  grade = 3:1,
  desc = c("Excellent", "Good", "Poor"),
  fail = c(F, F, T)
)

Then, let’s say we want to duplicate the info table so that we have a row for each value in grades. An elegant way to do this is by combining match() and integer subsetting (match(needles, haystack) returns the position where each needle is found in the haystack).

id <- match(grades, info$grade)
id
#> [1] 3 2 2 1 3
info[id, ]
#>     grade      desc fail
#> 3       1      Poor TRUE
#> 2       2      Good FALSE
#> 2.1     2      Good FALSE
#> 1       3 Excellent FALSE
#> 3.1     1      Poor TRUE

If you’re matching on multiple columns, you’ll need to first collapse them into a single column (with e.g. interaction()). Typically, however, you’re better off switching to a function designed specifically for joining multiple tables like merge(), or dplyr::left_join().

You can use integer indices to randomly sample or bootstrap a vector or data frame. Just use sample(n) to generate a random permutation of 1:n, and then use the results to subset the values:

df <- data.frame(x = c(1, 2, 3, 1, 2), y = 5:1, z = letters[1:5])

# Randomly reorder
df[sample(nrow(df)), ]
#>   x y z
#> 1 1 5 a
#> 4 1 2 d
#> 2 2 4 b
#> 5 2 1 e
#> 3 3 3 c

# Select 3 random rows
df[sample(nrow(df), 3), ]
#>   x y z
#> 3 3 3 c
#> 2 2 4 b

#> 1 1 5 a

# Select 6 bootstrap replicates
df[sample(nrow(df), 6, replace = TRUE), ]
#>     x y z
#> 4 1 2 d
#> 4.1 1 2 d
#> 5 2 1 e
#> 1 1 5 a
#> 1.1 1 5 a
#> 2   2 4 b

The arguments of sample() control the number of samples to extract, and also whether sampling is done with or without replacement.

order() takes a vector as its input and returns an integer vector describing how to order the subsetted vector²:

x <- c("b", "c", "a")
order(x)
#> [1] 3 1 2
x[order(x)]
#> [1] "a" "b" "c"

To break ties, you can supply additional variables to order(). You can also change the order from ascending to descending by using decreasing = TRUE. By default, any missing values will be put at the end of the vector; however, you can remove them with na.last = NA or put them at the front with na.last = FALSE.

For two or more dimensions, order() and integer subsetting makes it easy to order either the rows or columns of an object:

# Randomly reorder df
df2 <- df[sample(nrow(df)), 3:1]
df2
#>   z y x
#> 3 c 3 3

#>  1  a  5  1
#>  2  b  4  2
#>  4  d  2  1
#>  5  e  1  2

df2[order(df2$x), ]
#>   z y x
#> 1 a 5 1
#> 4 d 2 1
#> 2 b 4 2
#> 5 e 1 2
#> 3 c 3 3
df2[, order(names(df2))]
#>   x y z
#> 3 3 3 c
#> 1 1 5 a
#> 2 2 4 b
#> 4 1 2 d
#> 5 2 1 e

You can sort vectors directly with sort(), or similarly dplyr::arrange(), to sort a data frame.

Sometimes you get a data frame where identical rows have been collapsed into one and a count column has been added. rep() and integer subsetting make it easy to uncollapse, because we can take advantage of rep()s vectorisation: rep(x, y) repeats x[i] y[i] times.

df <- data.frame(x = c(2, 4, 1), y = c(9, 11, 6), n = c(3, 5, 1))
rep(1:nrow(df), df$n)
#> [1] 1 1 1 2 2 2 2 2 3

df[rep(1:nrow(df), df$n), ]
#>     x y n
#> 1   2 9 3
#> 1.1 2 9 3
#> 1.2 2 9 3
#> 2 4 11 5
#> 2.1 4 11 5
#> 2.2 4 11 5
#> 2.3 4 11 5

#> 2.4 4 11 5
#> 3   1 6 1

There are two ways to remove columns from a data frame. You can set individual columns to NULL:

df <- data.frame(x = 1:3, y = 3:1, z = letters[1:3])
df$z <- NULL

Or you can subset to return only the columns you want:

df <- data.frame(x = 1:3, y = 3:1, z = letters[1:3])
df[c("x", "y")]
#>   x y
#> 1 1 3
#> 2 2 2
#> 3 3 1

If you only know the columns you don’t want, use set operations to work out which columns to keep:

df[setdiff(names(df), "z")]
#>   x y
#> 1 1 3
#> 2 2 2
#> 3 3 1

Because logical subsetting allows you to easily combine conditions from multiple columns, it’s probably the most commonly used technique for extracting rows out of a data frame.

mtcars[mtcars$gear == 5, ]
#>                mpg cyl  disp  hp drat   wt qsec vs am gear carb
#> Porsche 914-2  26.0  4 120.3  91 4.43 2.14 16.7  0  1    5    2
#> Lotus Europa   30.4  4  95.1 113 3.77 1.51 16.9  1  1    5    2

#> Ford Pantera L 15.8  8 351.0 264 4.22 3.17 14.5  0  1     5    4
#> Ferrari Dino   19.7  6 145.0 175 3.62 2.77 15.5  0  1     5    6
#> Maserati Bora  15.0  8 301.0 335 3.54 3.57 14.6  0  1     5    8

mtcars[mtcars$gear == 5 & mtcars$cyl == 4, ]
#>               mpg cyl   disp hp  drat   wt qsec vs am  gear carb
#> Porsche 914-2 26.0 4   120.3 91  4.43 2.14 16.7  0  1     5    2
#> Lotus Europa 30.4 4     95.1 113 3.77 1.51 16.9  1  1     5    2

Remember to use the vector boolean operators & and |, not the short-circuiting scalar operators && and ||, which are more useful inside if statements. And don’t forget De Morgan’s laws (http://en.wikipedia.org/wiki/De_Morgan’s_laws), which can be useful to simplify negations:

• !(X & Y) is the same as !X | !Y

• !(X | Y) is the same as !X & !Y

For example, !(X & !(Y | Z)) simplifies to !X | !!(Y|Z), and then to !X | Y | Z.

It’s useful to be aware of the natural equivalence between set operations (integer subsetting) and Boolean algebra (logical subsetting). Using set operations is more effective when:

• You want to find the first (or last) TRUE.

• You have very few TRUEs and very many FALSEs; a set representation may be faster and require less storage.

which() allows you to convert a Boolean representation to an integer representation. There’s no reverse operation in base R but we can easily create one:

x <- sample(10) < 4
which(x)
#> [1] 2 5 8

unwhich <- function(x, n) {
  out <- rep_len(FALSE, n)
  out[x] <- TRUE
  out
}

unwhich(which(x), 10)
#>  [1] FALSE TRUE FALSE FALSE        TRUE FALSE FALSE    TRUE FALSE FALSE

Let’s create two logical vectors and their integer equivalents, and then explore the relationship between Boolean and set operations.

(x1 <- 1:10 %% 2 == 0)
#>  [1] FALSE  TRUE  FALSE  TRUE  FALSE  TRUE  FALSE  TRUE  FALSE  TRUE
(x2 <- which(x1))
#>  [1] 2 4 6 8 10
(y1 <- 1:10 %% 5 == 0)
#>  [1] FALSE  FALSE FALSE  FALSE TRUE   FALSE FALSE  FALSE FALSE  TRUE
(y2 <- which(y1))
#> [1] 5 10

# X & Y <-> intersect(x, y)
x1 & y1
#> [1] FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE FALSE TRUE
intersect(x2, y2)
#> [1] 10

# X | Y <-> union(x, y)
x1 | y1
#>  [1] FALSE TRUE FALSE       TRUE   TRUE   TRUE FALSE   TRUE FALSE   TRUE
union(x2, y2)
#>  [1] 2 4 6 8 10 5

# X & !Y <-> setdiff(x, y)
x1 & !y1
#>  [1] FALSE TRUE FALSE TRUE FALSE          TRUE FALSE   TRUE FALSE FALSE
setdiff(x2, y2)
#> [1] 2 4 6 8

# xor(X, Y) <-> setdiff(union(x, y), intersect(x, y))
xor(x1, y1)
#> [1] FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE
setdiff(union(x2, y2), intersect(x2, y2))
#> [1] 2 4 6 8 5

When first learning subsetting, a common mistake is to use x[which(y)] instead of x[y]. Here the which() achieves nothing: it switches from logical to integer subsetting but the result is exactly the same. In more general cases, there are two important differences.

• When the logical vector contains NA, logical subsetting replaces these values with NA while which() simply drops these values. It’s not uncommon to use which() for this side-effect, but I don’t recommend it: nothing about the name “which” implies the removal of missing values.

• x[-which(y)] is not equivalent to x[!y]: if y is all FALSE, which(y) will be integer(0) and -integer(0) is still integer(0), so you’ll get no values, instead of all values.

In general, avoid switching from logical to integer subsetting unless you want, for example, the first or last TRUE value.

1. How would you randomly permute the columns of a data frame? (This is an important technique in random forests.) Can you simultaneously permute the rows and columns in one step?

2. How would you select a random sample of m rows from a data frame? What if the sample had to be contiguous (i.e., with an initial row, a final row, and every row in between)?

3. How could you put the columns in a data frame in alphabetical order?

Want to skip this chapter? Go for it, if you can answer the questions below. Find the answers at the end of the chapter in Section 5.4.

• What is the different between if and ifelse()?

• In the following code, what will the value of y be if x is TRUE? What if x is FALSE? What if x is NA?

  y <- if (x) 3
  

• What does switch(”x”, x = , y = 2, z = 3) return?

• Section 5.2 dives into the details of if, then discusses the close relatives ifelse() and switch().

• Section 5.3 starts off by reminding you of the basic structure of the for loop in R, discusses some common pitfalls, and then talks about the related while and repeat statements.

The condition should evaluate to a single TRUE or FALSE. Most other inputs will generate an error:

if ("x") 1
#> Error in if   ("x") 1: argument is not interpretable as logical
if (logical())   1
#> Error in if   (logical()) 1: argument is of length zero
if (NA) 1
#> Error in if   (NA) 1: missing value where TRUE/FALSE needed

The exception is a logical vector of length greater than 1, which generates a warning:

if (c(TRUE, FALSE)) 1
#> Warning in if (c(TRUE, FALSE)) 1: the condition has length > 1 and
#> only the first element will be used
#> [1] 1

In R 3.5.0 and greater, thanks to Henrik Bengtsson (https://github.com/HenrikBengtsson/Wishlist-for-R/issues/38), you can turn this into an error by setting an environment variable:

Sys.setenv("_R_CHECK_LENGTH_1_CONDITION_" = "true")
if (c(TRUE, FALSE)) 1
#> Error in if (c(TRUE, FALSE)) 1: the condition has length > 1

I think this is good practice as it reveals a clear mistake that you might otherwise miss if it were only shown as a warning.

Given that if only works with a single TRUE or FALSE, you might wonder what to do if you have a vector of logical values. Handling vectors of values is the job of ifelse(): a vectorised function with test, yes, and no vectors (that will be recycled to the same length):

x <- 1:10
ifelse(x %% 5 == 0, "XXX", as.character(x))
#>  [1] "1"   "2"   "3"    "4"   "XXX" "6"   "7"   "8"   "9"   "XXX"

ifelse(x %% 2 == 0, "even", "odd")
#>  [1] "odd" "even" "odd" "even" "odd" "even" "odd" "even" "odd"
#> [10] "even"

Note that missing values will be propagated into the output.

I recommend using ifelse() only when the yes and no vectors are the same type as it is otherwise hard to predict the output type. See about https://vctrs.r-lib.org/articles/stability.html#ifelse for additional discussion.

Another vectorised equivalent is the more general dplyr::case_when(). It uses a special syntax to allow any number of condition-vector pairs:

dplyr::case_when(
  x %% 35 == 0 ~ "fizz buzz",
  x %% 5 == 0 ~ "fizz",
  x %% 7 == 0 ~ "buzz",
  is.na(x) ~ "???",
  TRUE ~ as.character(x)
)
#> [1] "1"     "2"    "3"    "4"        "fizz" "6"      "buzz" "8"     "9"
#> [10] "fizz"

Closely related to if is the switch()-statement. It’s a compact, special purpose equivalent that lets you replace code like:

x_option <- function(x) {
  if (x == "a") {
    "option 1"
  } else if (x == "b") {

    "option 2"
  } else if (x == "c") {
    "option 3"
  } else {
    stop("Invalid `x` value")
  }
}

with the more succinct:

x_option <- function(x) {
  switch(x,
    a = "option 1",
    b = "option 2",
    c = "option 3",
    stop("Invalid `x` value")
  )
}

The last component of a switch() should always throw an error, otherwise unmatched inputs will invisibly return NULL:

(switch("c", a = 1, b = 2))
#> NULL

If multiple inputs have the same output, you can leave the right hand side of = empty and the input will “fall through” to the next value. This mimics the behaviour of C’s switch statement:

legs <- function(x) {
  switch(x,
    cow = ,
    horse = ,
    dog = 4,
    human = ,
    chicken = 2,
    plant = 0,
    stop("Unknown input")
  )
}
legs("cow")
#> [1] 4

legs("dog")
#> [1] 4

It is also possible to use switch() with a numeric x, but is harder to read, and has undesirable failure modes if x is a not a whole number. I recommend using switch() only with character inputs.

1. What type of vector does each of the following calls to ifelse() return?

   ifelse(TRUE, 1, "no")
   ifelse(FALSE, 1, "no")
   ifelse(NA, 1, "no")
   

   Read the documentation and write down the rules in your own words.

2. Why does the following code work?

   x <- 1:10
   if (length(x)) "not empty" else "empty"
   #> [1] "not empty"
   
   x <- numeric()
   if (length(x)) "not empty" else "empty"
   #> [1] "empty"
   

There are three common pitfalls to watch out for when using for. First, if you’re generating data, make sure to preallocate the output container. Otherwise the loop will be very slow; see Sections 23.2.2 and 24.6 for more details. The vector() function is helpful here.

means <- c(1, 50, 20)
out <- vector("list", length(means))
for (i in 1:length(means)) {
  out[[i]] <- rnorm(10, means[[i]])
}

Next, beware of iterating over 1:length(x), which will fail in unhelpful ways if x has length 0:

means <- c()
out <- vector("list", length(means))
for (i in 1:length(means)) {
  out[[i]] <- rnorm(10, means[[i]])
}
#> Error in rnorm(10, means[[i]]): invalid arguments

This occurs because : works with both increasing and decreasing sequences:

1:length(means)
#> [1] 1 0

Use seq_along(x) instead. It always returns a value the same length as x:

seq_along(means)
#> integer(0)

out <- vector("list", length(means))
for (i in seq_along(means)) {
  out[[i]] <- rnorm(10, means[[i]])
}

Finally, you might encounter problems when iterating over S3 vectors, as loops typically strip the attributes:

xs <- as.Date(c("2020-01-01", "2010-01-01"))
for (x in xs) {
  print(x)
}
#> [1] 18262
#> [1] 14610

Work around this by calling [[ yourself:

for (i in seq_along(xs)) {
  print(xs[[i]])
}
#> [1] "2020-01-01"
#> [1] "2010-01-01"

for loops are useful if you know in advance the set of values that you want to iterate over. If you don’t know, there are two related tools with more flexible specifications:

• while(condition) action: performs action while condition is TRUE.

• repeat(action): repeats action forever (i.e. until it encounters break).

R does not have an equivalent to the do {action} while (condition) syntax found in other languages.

You can rewrite any for loop to use while instead, and you can rewrite any while loop to use repeat, but the converses are not true. That means while is more flexible than for, and repeat is more flexible than while. It’s good practice, however, to use the least-flexible solution to a problem, so you should use for wherever possible.

Generally speaking you shouldn’t need to use for loops for data analysis tasks, as map() and apply() already provide less flexible solutions to most problems. You’ll learn more in Chapter 9.

1. Why does this code succeed without errors or warnings?

   x <- numeric()
   out <- vector("list", length(x))
   for (i in 1:length(x)) {
     out[i] <- x[i] ^ 2
   }
   out
   

2. When the following code is evaluated, what can you say about the vector being iterated?

   xs <- c(1, 2, 3)
   for (x in xs) {
      xs <- c(xs, x * 2)
   }
   xs
   #> [1] 1 2 3 2 4 6
   

3. What does the following code tell you about when the index is updated?

   for (i in 1:3) {
     i <- i * 2
     print(i)
   }
   #> [1] 2
   #> [1] 4
   #> [1] 6
   

Answer the following questions to see if you can safely skip this chapter. You can find the answers in Section 6.9.

1. What are the three components of a function?

2. What does the following code return?

   x <- 10
   f1 <- function(x) {
     function() {
       x + 10
     }
   }
   f1(1)()
   

3. How would you usually write this code?

   `+`(1, `*`(2, 3))
   

4. How could you make this call easier to read?

   mean(, TRUE, x = c(1:10, NA))
   

5. Does the following code throw an error when executed? Why or why not?

   f2 <- function(a, b) {
     a * 10
   }
   f2(10, stop("This is an error!"))
   

6. What is an infix function? How do you write it? What’s a replacement function? How do you write it?

7. How do you ensure that cleanup action occurs regardless of how a function exits?

• Section 6.2 describes the basics of creating a function, the three main components of a function, and the exception to many function rules: primitive functions (which are implemented in C, not R).

• Section 6.3 discusses the strengths and weaknesses of the three forms of function composition commonly used in R code.

• Section 6.4 shows you how R finds the value associated with a given name, i.e. the rules of lexical scoping.

• Section 6.5 is devoted to an important property of function arguments: they are only evaluated when used for the first time.

• Section 6.6 discusses the special … argument, which allows you to pass on extra arguments to another function.

• Section 6.7 discusses the two primary ways that a function can exit, and how to define an exit handler, code that is run on exit, regardless of what triggers it.

• Section 6.8 shows you the various ways in which R disguises ordinary function calls, and how you can use the standard prefix form to better understand what’s going on.

A function has three parts:

• The formals(), the list of arguments that control how you call the function.

• The body(), the code inside the function.

• The environment(), the data structure that determines how the function finds the values associated with the names.

While the formals and body are specified explicitly when you create a function, the environment is specified implicitly, based on where you defined the function. The function environment always exists, but it is only printed when the function isn’t defined in the global environment.

f02 <- function(x, y) {
  # A comment
  x + y
}

formals(f02)
#> $x
#>
#>
#> $y

body(f02)
#> {
#>     x + y
#> }

environment(f02)
#> <environment: R_GlobalEnv>

I’ll draw functions as in the following diagram. The black dot on the left is the environment. The two blocks to the right are the function arguments. I won’t draw the body, because it’s usually large, and doesn’t help you understand the shape of the function.

Like all objects in R, functions can also possess any number of additional attributes(). One attribute used by base R is srcref, short for source reference. It points to the source code used to create the function. The srcref is used for printing because, unlike body(), it contains code comments and other formatting.

attr(f02, "srcref")
#> function(x, y) {
#>   # A comment
#>   x + y
#> }

There is one exception to the rule that a function has three components. Primitive functions, like sum() and [, call C code directly.

sum
#> function (..., na.rm = FALSE)      .Primitive("sum")
`[`
#> .Primitive("[")

They have either type builtin or type special.

typeof(sum)
#> [1] "builtin"
typeof(`[`)
#> [1] "special"

These functions exist primarily in C, not R, so their formals(), body(), and environment() are all NULL:

formals(sum)
#> NULL
body(sum)
#> NULL
environment(sum)
#> NULL

Primitive functions are only found in the base package. While they have certain performance advantages, this benefit comes at a price: they are harder to write. For this reason, R-core generally avoids creating them unless there is no other option.

It’s very important to understand that R functions are objects in their own right, a language property often called “first-class functions”. Unlike in many other languages, there is no special syntax for defining and naming a function: you simply create a function object (with function) and bind it to a name with <-:

f01 <- function(x) {
  sin(1 / x ^ 2)
}

While you almost always create a function and then bind it to a name, the binding step is not compulsory. If you choose not to give a function a name, you get an anonymous function. This is useful when it’s not worth the effort to figure out a name:

lapply(mtcars, function(x) length(unique(x)))
Filter(function(x) !is.numeric(x), mtcars)
integrate(function(x) sin(x) ^ 2, 0, pi)

A final option is to put functions in a list:

funs <- list(
  half = function(x) x / 2,
  double = function(x) x * 2
)

funs$double(10)
#> [1] 20

In R, you’ll often see functions called closures. This name reflects the fact that R functions capture, or enclose, their environments, which you’ll learn more about in Section 7.4.2.

You normally call a function by placing with arguments wrapped in parentheses: mean(1:10, na.rm = TRUE). But what happens if you have the arguments already in a data structure?

args <- list(1:10, na.rm = TRUE)

You can instead use do.call(): it has two arguments. The function to call, and a list containing the function arguments:

do.call(mean, args)
#> [1] 5.5

We’ll come back to this idea in Section 19.6.

1. Given a name, like ”mean”, match.fun() lets you find a function. Given a function, can you find its name? Why doesn’t that make sense in R?

2. It’s possible (although typically not useful) to call an anonymous function. Which of the two approaches below is correct? Why?

   function(x) 3()
   #> function(x) 3()
   (function(x) 3)()
   #> [1] 3
   

3. A good rule of thumb is that an anonymous function should fit on one line and shouldn’t need to use {}. Review your code. Where could you have used an anonymous function instead of a named function? Where should you have used a named function instead of an anonymous function?

4. What function allows you to tell if an object is a function? What function allows you to tell if a function is a primitive function?

5. This code makes a list of all functions in the base package.

   objs <- mget(ls("package:base", all = TRUE), inherits = TRUE)
   funs <- Filter(is.function, objs)
   

   Use it to answer the following questions:

   1. Which base function has the most arguments?

   2. How many base functions have no arguments? What’s special about those functions?

   3. How could you adapt the code to find all primitive functions?

6. What are the three important components of a function?

7. When does printing a function not show the environment it was created in?

The basic principle of lexical scoping is that names defined inside a function mask names defined outside a function. This is illustrated in the following example.

x <- 10
y <- 20
g02 <- function() {
  x <- 1
  y <- 2
  c(x, y)
}
g02()
#> [1] 1 2

If a name isn’t defined inside a function, R looks one level up.

x <- 2
g03 <-   function() {
  y <-   1
  c(x,   y)
}
g03()
#> [1]   2 1

# And this doesn't change the previous value of y
y
#> [1] 20

The same rules apply if a function is defined inside another function. First, R looks inside the current function. Then, it looks where that function was defined (and so on, all the way up to the global environment). Finally, it looks in other loaded packages.

Run the following code in your head, then confirm the result by running the code.³

x <- 1
g04 <- function() {
  y <- 2
  i <- function() {
    z <- 3
    c(x, y, z)
  }
  i()
}
g04()

The same rules also apply to functions created by other functions, which I call manufactured functions, the topic of Chapter 10.

In R, functions are ordinary objects. This means the scoping rules described above also apply to functions:

g07 <- function(x) x + 1
g08 <- function() {
  g07 <- function(x) x + 100
  g07(10)
}
g08()
#> [1] 110

However, when a function and a non-function share the same name (they must, of course, reside in different environments), applying these rules gets a little more complicated. When you use a name in a function call, R ignores non-function objects when looking for that value. For example, in the code below, g09 takes on two different values:

g09<- function(x) x + 100
g10<- function() {
  g09 <- 10
  g09(g09)
}
g10()
#> [1] 110

For the record, using the same name for different things is confusing and best avoided!

What happens to values between invocations of a function? Consider the example below. What will happen the first time you run this function? What will happen the second time?⁴ (If you haven’t seen exists() before, it returns TRUE if there’s a variable with that name and returns FALSE if not.)

g11 <- function() {
  if (!exists("a")) {
    a <- 1
  } else {
    a <- a + 1
  }
  a
}

g11()
g11()

You might be surprised that g11() always returns the same value. This happens because every time a function is called a new environment is created to host its execution. This means that a function has no way to tell what happened the last time it was run; each invocation is completely independent. We’ll see some ways to get around this in Section 10.2.4.

Lexical scoping determines where, but not when to look for values. R looks for values when the function is run, not when the function is created. Together, these two properties tell us that the output of a function can differ depending on the objects outside the function’s environment:

g12 <- function() x + 1
x <- 15
g12()
#> [1] 16

x <- 20
g12()
#> [1] 21

This behaviour can be quite annoying. If you make a spelling mistake in your code, you won’t get an error message when you create the function. And depending on the variables defined in the global environment, you might not even get an error message when you run the function.

To detect this problem, use codetools::findGlobals(). This function lists all the external dependencies (unbound symbols) within a function:

codetools::findGlobals(g12)
#> [1] "+" "x"

To solve this problem, you can manually change the function’s environment to the emptyenv(), an environment which contains nothing:

environment(g12) <- emptyenv()
g12()
#> Error in x + 1: could not find function "+"

The problem and its solution reveal why this seemingly undesirable behaviour exists: R relies on lexical scoping to find everything, from the obvious, like mean(), to the less obvious, like + or even {. This gives R’s scoping rules a rather beautiful simplicity.

1. What does the following code return? Why? Describe how each of the three c’s is interpreted.

   c <- 10
   c(c = c)
   

2. What are the four principles that govern how R looks for values?

3. What does the following function return? Make a prediction before running the code yourself.

   f <- function(x) {
     f <- function(x) {
       f <- function() {
         x ^ 2
       }
       f() + 1
   
     }
     f(x) * 2
   }
   f(10)
   

Lazy evaluation is powered by a data structure called a promise, or (less commonly) a thunk. It’s one of the features that makes R such an interesting programming language (we’ll return to promises again in Section 20.3).

A promise has three components:

• An expression, like x + y, which gives rise to the delayed computation.

• An environment where the expression should be evaluated, i.e. the environment where the function is called. This makes sure that the following function returns 11, not 101:

  y <- 10
  h02 <- function(x) {
    y <- 100
    x + 1
  
  }
  
  h02(y)
  #> [1] 11
  

  This also means that when you do assignment inside a call to a function, the variable is bound outside of the function, not inside of it.

  h02(y <- 1000)
  #> [1] 1001
  y
  #> [1] 1000
  

• A value, which is computed and cached the first time a promise is accessed when the expression is evaluated in the specified environment. This ensures that the promise is evaluated at most once, and is why you only see “Calculating…” printed once in the following example.

  double <- function(x) {
    message("Calculating...")
    x * 2
  }
  
  h03 <- function(x) {
    c(x, x)
  }
  
  h03(double(x))
  #> Calculating...
  #> [1] 40 40
  

You cannot manipulate promises with R code. Promises are like a quantum state: any attempt to inspect them with R code will force an immediate evaluation, making the promise disappear. Later, in Section 20.3, you’ll learn about quosures, which convert promises into an R object where you can easily inspect the expression and the environment.

Thanks to lazy evaluation, default values can be defined in terms of other arguments, or even in terms of variables defined later in the function:

h04 <- function(x = 1, y = x * 2, z = a + b) {
  a <- 10
  b <- 100

  c(x, y, z)
}

h04()
#> [1]    1      2 110

Many base R functions use this technique, but I don’t recommend it. It makes the code harder to understand: to predict what will be returned, you need to know the exact order in which default arguments are evaluated.

The evaluation environment is slightly different for default and user supplied arguments, as default arguments are evaluated inside the function. This means that seemingly identical calls can yield different results. It’s easiest to see this with an extreme example:

h05 <- function(x = ls()) {
  a <- 1
  x
}

# ls() evaluated inside h05:
h05()
#> [1] "a" "x"

# ls() evaluated in global environment:
h05(ls())
#> [1] "h05"

To determine if an argument’s value comes from the user or from a default, you can use missing():

h06 <- function(x = 10) {
  list(missing(x), x)
}
str(h06())
#> List of 2

#> $ : logi TRUE
#> $ : num 10
str(h06(10))
#> List of 2
#> $ : logi FALSE
#> $ : num 10

missing() is best used sparingly, however. Take sample(), for example. How many arguments are required?

args(sample)
#> function (x, size, replace = FALSE, prob = NULL)
#> NULL

It looks like both x and size are required, but if size is not supplied, sample() uses missing() to provide a default. If I were to rewrite sample, I’d use an explicit NULL to indicate that size is not required but can be supplied:

sample <- function(x, size = NULL, replace = FALSE, prob = NULL) {
  if (is.null(size)) {
    size <- length(x)
  }

  x[sample.int(length(x), size, replace = replace, prob = prob)]
}

With the binary pattern created by the %||% infix function, which uses the left side if it’s not NULL and the right side otherwise, we can further simplify sample():

`%||%` <- function(lhs, rhs) {
  if (!is.null(lhs)) {
    lhs
  } else {
    rhs
  }
}

sample <- function(x, size = NULL, replace = FALSE, prob = NULL) {
  size <- size %||% length(x)
  x[sample.int(length(x), size, replace = replace, prob = prob)]
}

Because of lazy evaluation, you don’t need to worry about unnecessary computation: the right side of %||% will only be evaluated if the left side is NULL.

1. What important property of && makes x_ok() work?

   x_ok <- function(x) {
     !is.null(x) && length(x) == 1 && x > 0
   }
   
   x_ok(NULL)
   #> [1] FALSE
   x_ok(1)
   #> [1] TRUE
   x_ok(1:3)
   #> [1] FALSE
   

   What is different with this code? Why is this behaviour undesirable here?

   x_ok <- function(x) {
     !is.null(x) & length(x) == 1 & x > 0
   }
   
   x_ok(NULL)
   #> logical(0)
   x_ok(1)
   #> [1] TRUE
   x_ok(1:3)
   #> [1] FALSE FALSE FALSE
   

2. What does this function return? Why? Which principle does it illustrate?

   f2 <- function(x = z) {
     z <- 100
     x
   }
   f2()
   

3. What does this function return? Why? Which principle does it illustrate?

   y <- 10
   f1 <- function(x = {y <- 1; 2}, y = 0) {
     c(x, y)
   }
   f1()
   y
   

4. In hist(), the default value of xlim is range(breaks), the default value for breaks is ”Sturges”, and

   range("Sturges")
   #> [1] "Sturges" "Sturges"
   

   Explain how hist() works to get a correct xlim value.

5. Explain why this function works. Why is it confusing?

   show_time <- function(x = stop("Error!")) {
     stop <- function(...) Sys.time()
     print(x)
   }
   show_time()
   #> [1] "2019-04-04 11:48:56 CDT"
   

6. How many arguments are required when calling library()?

1. Explain the following results:

   sum(1, 2, 3)
   #> [1] 6
   mean(1, 2, 3)
   #> [1] 1
   
   sum(1, 2, 3, na.omit = TRUE)
   #> [1] 7
   
   mean(1, 2, 3, na.omit = TRUE)
   #> [1] 1
   

2. In the following call, explain how to find the documentation for the named arguments in the following function call:

   plot(1:10, col = "red", pch = 20, xlab = "x", col.lab = "blue")
   

   

3. Why does plot(1:10, col = ”red”) only colour the points, not the axes or labels? Read the source code of plot.default() to find out.

There are two ways that a function can return a value:

• Implicitly, where the last evaluated expression is the return value:

  j01 <- function(x) {
    if (x < 10) {
      0
    } else {
      10
    }
  }
  j01(5)
  #> [1] 0
  j01(15)
  #> [1] 10
  

• Explicitly, by calling return():

  j02 <- function(x) {
    if (x < 10) {
      return(0)
    } else {
      return(10)
    }
  }
  

Most functions return visibly: calling the function in an interactive context prints the result.

j03 <- function() 1
j03()
#> [1] 1

However, you can prevent automatic printing by applying invisible() to the last value:

j04 <- function() invisible(1)
j04()

To verify that this value does indeed exist, you can explicitly print it or wrap it in parentheses:

print(j04())
#> [1] 1

(j04())
#> [1] 1

Alternatively, you can use withVisible() to return the value and a visibility flag:

str(withVisible(j04()))
#> List of 2
#>  $ value : num 1
#>  $ visible: logi FALSE

The most common function that returns invisibly is <-:

a <- 2
(a <- 2)
#> [1] 2

This is what makes it possible to chain assignments:

a <- b <- c <- d <- 2

In general, any function called primarily for a side effect (like <-, print(), or plot()) should return an invisible value (typically the value of the first argument).

If a function cannot complete its assigned task, it should throw an error with stop(), which immediately terminates the execution of the function.

j05 <- function() {
  stop("I'm an error")
  return(10)
}
j05()
#> Error in j05(): I'm an error

An error indicates that something has gone wrong, and forces the user to deal with the problem. Some languages (like C, Go, and Rust) rely on special return values to indicate problems, but in R you should always throw an error. You’ll learn more about errors, and how to handle them, in Conditions.

Sometimes a function needs to make temporary changes to the global state. But having to cleanup those changes can be painful (what happens if there’s an error?). To ensure that these changes are undone and that the global state is restored no matter how a function exits, use on.exit() to set up an exit handler. The following simple example shows that the exit handler is run regardless of whether the function exits normally or with an error.

j06 <- function(x) {
  cat("Hello\n")
  on.exit(cat("Goodbye!\n"), add = TRUE)

  if (x) {
    return(10)
  } else {
    stop("Error")
  }
}

j06(TRUE)
#> Hello
#> Goodbye!
#> [1] 10

j06(FALSE)
#> Hello
#> Error in j06(FALSE): Error
#> Goodbye!

on.exit() is useful because it allows you to place clean-up code directly next to the code that requires clean-up:

cleanup <- function(dir, code) {
  old_dir <- setwd(dir)
  on.exit(setwd(old_dir), add = TRUE)

  old_opt <- options(stringsAsFactors = FALSE)
  on.exit(options(old_opt), add = TRUE)
}

Coupled with lazy evaluation, this creates a very useful pattern for running a block of code in an altered environment:

with_dir <- function(dir, code) {
  old <- setwd(dir)
  on.exit(setwd(old), add = TRUE)

  force(code)
}

getwd()
#> [1] "/Users/hadley/Documents/adv-r/adv-r"
with_dir("~", getwd())
#> [1] "/Users/hadley"

The use of force() isn’t strictly necessary here as simply referring to code will force its evaluation. However, using force() makes it very clear that we are deliberately forcing the execution. You’ll learn other uses of force() in Chapter 10.

The withr package [Hester et al., 2018] provides a collection of other functions for setting up a temporary state.

In R 3.4 and earlier, on.exit() expressions are always run in order of creation:

j08 <- function() {
  on.exit(message("a"), add = TRUE)
  on.exit(message("b"), add = TRUE)
}
j08()
#> a
#> b

This can make cleanup a little tricky if some actions need to happen in a specific order; typically you want the most recent added expression to be run first. In R 3.5 and later, you can control this by setting after = FALSE:

j09 <- function() {
  on.exit(message("a"), add = TRUE, after = FALSE)
  on.exit(message("b"), add = TRUE, after = FALSE)
}
j09()
#> b
#> a

1. What does load() return? Why don’t you normally see these values?

2. What does write.table() return? What would be more useful?

3. How does the chdir parameter of source() compare to in_dir()? Why might you prefer one to the other?

4. Write a function that opens a graphics device, runs the supplied code, and closes the graphics device (always, regardless of whether or not the plotting code works).

5. We can use on.exit() to implement a simple version of capture.output().

   capture.output2 <- function(code) {
     temp <- tempfile()
     on.exit(file.remove(temp), add = TRUE, after = TRUE)
   
     sink(temp)
     on.exit(sink(), add = TRUE, after = TRUE)
   
     force(code)
     readLines(temp)
   }
   capture.output2(cat("a", "b", "c", sep = "\n"))
   #> [1] "a" "b" "c"
   

   Compare capture.output() to capture.output2(). How do the functions differ? What features have I removed to make the key ideas easier to see? How have I rewritten the key ideas so they’re easier to understand?

An interesting property of R is that every infix, replacement, or special form can be rewritten in prefix form. Doing so is useful because it helps you better understand the structure of the language, it gives you the real name of every function, and it allows you to modify those functions for fun and profit.

The following example shows three pairs of equivalent calls, rewriting an infix form, replacement form, and a special form into prefix form.

x + y
`+`(x, y)

names(df) <- c("x", "y", "z")
`names<-`(df, c("x", "y", "z"))

for(i in 1:10) print(i)
`for`(i, 1:10, print(i))

Suprisingly, in R, for can be called like a regular function! The same is true for basically every operation in R, which means that knowing the function name of a non-prefix function allows you to override its behaviour. For example, if you’re ever feeling particularly evil, run the following code while a friend is away from their computer. It will introduce a fun bug: 10% of the time, it will add 1 to any numeric calculation inside the parentheses.

`(` <- function(e1) {
  if (is.numeric(e1) && runif(1) < 0.1) {
    e1 + 1
  } else {
    e1
  }
}
replicate(50, (1 + 2))
#> [1] 3 3 3 3 3 3 3 3 3 3 3 3 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
#> [33] 3 3 3 3 3 3 4 3 4 3 3 3 3 4 3 3 3 3
rm("(")

Of course, overriding built-in functions like this is a bad idea, but, as you’ll learn in Section 21.2.5, it’s possible to apply it only to selected code blocks. This provides a clean and elegant approach to writing domain specific languages and translators to other languages.

A more useful application comes up when using functional programming tools. For example, you could use lapply() to add 3 to every element of a list by first defining a function add():

add <- function(x, y) x + y
lapply(list(1:3, 4:5), add, 3)
#> [[1]]
#> [1] 4 5 6
#>
#> [[2]]
#> [1] 7 8

But we can also get the same result simply by relying on the existing + function:

lapply(list(1:3, 4:5), `+`, 3)
#> [[1]]
#> [1] 4 5 6
#>
#> [[2]]
#> [1] 7 8

We’ll explore this idea in detail in Section 9.

The prefix form is the most common form in R code, and indeed in the majority of programming languages. Prefix calls in R are a little special because you can specify arguments in three ways:

• By position, like help(mean).

• Using partial matching, like help(top = mean).

• By name, like help(topic = mean).

As illustrated by the following chunk, arguments are matched by exact name, then with unique prefixes, and finally by position.

k01 <- function(abcdef, bcde1, bcde2) {
  list(a = abcdef, b1 = bcde1, b2 = bcde2)
}
str(k01(1, 2, 3))
#> List of 3
#> $ a : num 1
#> $ b1: num 2
#> $ b2: num 3
str(k01(2, 3, abcdef = 1))
#> List of 3
#> $ a : num 1
#> $ b1: num 2
#> $ b2: num 3

# Can abbreviate long argument names:
str(k01(2, 3, a = 1))
#> List of 3
#> $ a : num 1
#> $ b1: num 2
#> $ b2: num 3

# But this doesn't work because abbreviation is ambiguous

str(k01(1, 3, b = 1))
#> Error in k01(1, 3, b = 1): argument 3 matches multiple formal
#> arguments

In general, use positional matching only for the first one or two arguments; they will be the most commonly used, and most readers will know what they are. Avoid using positional matching for less commonly used arguments, and never use partial matching. Unfortunately you can’t disable partial matching, but you can turn it into a warning with the warnPartialMatchArgs option:

options(warnPartialMatchArgs = TRUE)
x <- k01(a = 1, 2, 3)
#> Warning in k01(a = 1, 2, 3): partial argument match of 'a' to
#> 'abcdef'

Infix functions get their name from the fact the function name comes inbetween its arguments, and hence have two arguments. R comes with a number of built-in infix operators: :, ::, :::, $, @, ^, *, /, +, -, >, >=, <, <=, ==, !=, !, &, &&, |, ||, ~, <-, and <<-. You can also create your own infix functions that start and end with %. Base R uses this pattern to define %%, %*%, %/%, %in%, %o%, and %x%.

Defining your own infix function is simple. You create a two argument function and bind it to a name that starts and ends with %:

`%+%` <- function(a, b) paste0(a, b)
"new " %+% "string"
#> [1] "new string"

The names of infix functions are more flexible than regular R functions: they can contain any sequence of characters except for %. You will need to escape any special characters in the string used to define the function, but not when you call it:

`% %` <- function(a, b) paste(a, b)
`%/\\%` <- function(a, b) paste(a, b)

"a" % % "b"
#> [1] "a b"

"a" %/\% "b"
#> [1] "a b"

R’s default precedence rules mean that infix operators are composed left to right:

`%-%` <- function(a, b) paste0("(", a, " %-% ", b, ")")
"a" %-% "b" %-% "c"
#> [1] "((a %-% b) %-% c)"

There are two special infix functions that can be called with a single argument: + and -.

-1
#> [1] -1
+10
#> [1] 10

Replacement functions act like they modify their arguments in place, and have the special name xxx<-. They must have arguments named x and value, and must return the modified object. For example, the following function modifies the second element of a vector:

`second<-` <- function(x, value) {
  x[2] <- value
  x
}

Replacement functions are used by placing the function call on the left side of <-:

x <- 1:10
second(x) <- 5L
x
#> [1] 1  5  3  4  5  6  7  8  9 10

I say they act like they modify their arguments in place, because, as explained in Section 2.5, they actually create a modified copy. We can see that by using tracemem():

x <- 1:10
tracemem(x)
#> <0x7ffae71bd880>

second(x) <- 6L
#> tracemem[0x7ffae71bd880 -> 0x7ffae61b5480]:
#> tracemem[0x7ffae61b5480 -> 0x7ffae73f0408]: second<-

If your replacement function needs additional arguments, place them between x and value, and call the replacement function with additional arguments on the left:

`modify<-` <- function(x, position, value) {
  x[position] <- value
  x
}
modify(x, 1) <- 10
x
#> [1] 10 5 3 4 5 6 7 8 9 10

When you write modify(x, 1) <- 10, behind the scenes R turns it into:

x <- `modify<-`(x, 1, 10)

Combining replacement with other functions requires more complex translation. For example:

x <- c(a = 1, b = 2, c = 3)
names(x)
#> [1] "a" "b" "c"

names(x)[2] <- "two"
names(x)
#> [1] "a" "two" "c"

is translated into:

`*tmp*` <- x
x <- `names<-`(`*tmp*`, `[<-`(names(`*tmp*`), 2, "two"))
rm(`*tmp*`)

(Yes, it really does create a local variable named *tmp*, which is removed afterwards.)

Finally, there are a bunch of language features that are usually written in special ways, but also have prefix forms. These include parentheses:

• (x) (`(`(x))

• {x} (`{`(x)).

The subsetting operators:

• x[i] (`[`(x, i))

• x[[i]] (`[[`(x, i))

And the tools of control flow:

• if (cond) true (`if`(cond, true))

• if (cond) true else false (`if`(cond, true, false))

• for(var in seq) action (`for`(var, seq, action))

• while(cond) action (`while`(cond, action))

• repeat expr (`repeat`(expr))

• next (`next`())

• break (`break`())

Finally, the most complex is the function function:

• function(arg1, arg2) {body} (`function`(alist(arg1, arg2), body, env))

Knowing the name of the function that underlies a special form is useful for getting documentation: ?( is a syntax error; ?`(` will give you the documentation for parentheses.

All special forms are implemented as primitive functions (i.e. in C); this means printing these functions is not informative:

`for`
#> .Primitive("for")

1. Rewrite the following code snippets into prefix form:

   1 + 2 + 3
   
   1 + (2 + 3)
   
   if (length(x) <= 5) x[[5]] else x[[n]]
   

2. Clarify the following list of odd function calls:

   x <- sample(replace = TRUE, 20, x = c(1:10, NA))
   y <- runif(min = 0, max = 1, 20)
   cor(m = "k", y = y, u = "p", x = x)
   

3. Explain why the following code fails:

   modify(get("x"), 1) <- 10
   #> Error: target of assignment expands to non-language object
   

4. Create a replacement function that modifies a random location in a vector.

5. Write your own version of + that pastes its inputs together if they are character vectors but behaves as usual otherwise. In other words, make this code work:

   1 + 2
   #> [1] 3
   
   "a" + "b"
   #> [1] "ab"
   

6. Create a list of all the replacement functions found in the base package. Which ones are primitive functions? (Hint: use apropos().)

7. What are valid names for user-created infix functions?

8. Create an infix xor() operator.

9. Create infix versions of the set functions intersect(), union(), and setdiff(). You might call them %n%, %u%, and %/% to match conventions from mathematics.

If you can answer the following questions correctly, you already know the most important topics in this chapter. You can find the answers at the end of the chapter in Section 7.7.

1. List at least three ways that an environment differs from a list.

2. What is the parent of the global environment? What is the only environment that doesn’t have a parent?

3. What is the enclosing environment of a function? Why is it important?

4. How do you determine the environment from which a function was called?

5. How are <- and <<- different?

• Section 7.2 introduces you to the basic properties of an environment and shows you how to create your own.

• Section 7.3 provides a function template for computing with environments, illustrating the idea with a useful function.

• Section 7.4 describes environments used for special purposes: for packages, within functions, for namespaces, and for function execution.

• Section 7.5 explains the last important environment: the caller environment. This requires you to learn about the call stack, that describes how a function was called. You’ll have seen the call stack if you’ve ever called traceback() to aid debugging.

• Section 7.6 briefly discusses three places where environments are useful data structures for solving other problems.

This chapter will use rlang (https://rlang.r-lib.org) functions for working with environments, because it allows us to focus on the essence of environments, rather than the incidental details.

library(rlang)

The env_ functions in rlang are designed to work with the pipe: all take an environment as the first argument, and many also return an environment. I won’t use the pipe in this chapter in the interest of keeping the code as simple as possible, but you should consider it for your own code.

To create an environment, use rlang::env(). It works like list(), taking a set of name-value pairs:

e1 <- env(
  a = FALSE,
  b = "a",
  c = 2.3,
  d = 1:3,
)

The job of an environment is to associate, or bind, a set of names to a set of values. You can think of an environment as a bag of names, with no implied order (i.e. it doesn’t make sense to ask which is the first element in an environment). For that reason, we’ll draw the environment as so:

As discussed in Section 2.5.2, environments have reference semantics: unlike most R objects, when you modify them, you modify them in place, and don’t create a copy. One important implication is that environments can contain themselves.

e1$d <- e1

Printing an environment just displays its memory address, which is not terribly useful:

e1
#> <environment: 0x7fba6eac4b08>

Instead, we’ll use env_print() which gives us a little more information:

env_print(e1)
#> <environment: 0x7fba6eac4b08>
#> parent: <environment: global>
#> bindings:
#>  * a: <lgl>
#>  * b: <chr>
#>  * c: <dbl>
#>  * d: <env>

You can use env_names() to get a character vector giving the current bindings

env_names(e1)
#> [1] "a" "b" "c" "d"

We’ll talk in detail about special environments in 7.4, but for now we need to mention two. The current environment, or current_env() is the environment in which code is currently executing. When you’re experimenting interactively, that’s usually the global environment, or global_env(). The global environment is sometimes called your “workspace”, as it’s where all interactive (i.e. outside of a function) computation takes place.

To compare environments, you need to use identical() and not ==. This is because == is a vectorised operator, and environments are not vectors.

identical(global_env(), current_env())
#> [1] TRUE

global_env() == current_env()
#> Error in global_env() == current_env(): comparison (1) is possible
#> only for atomic and list types

Every environment has a parent, another environment. In diagrams, the parent is shown as a small pale blue circle and arrow that points to another environment. The parent is what’s used to implement lexical scoping: if a name is not found in an environment, then R will look in its parent (and so on). You can set the parent environment by supplying an unnamed argument to env(). If you don’t supply it, it defaults to the current environment. In the code below, e2a is the parent of e2b.

e2a <- env(d = 4, e = 5)
e2b <- env(e2a, a = 1, b = 2, c = 3)

To save space, I typically won’t draw all the ancestors; just remember whenever you see a pale blue circle, there’s a parent environment somewhere.

You can find the parent of an environment with env_parent():

env_parent(e2b)
#> <environment: 0x7fba74178f48>
env_parent(e2a)
#> <environment: R_GlobalEnv>

Only one environment doesn’t have a parent: the empty environment. I draw the empty environment with a hollow parent environment, and where space allows I’ll label it with R_EmptyEnv, the name R uses.

e2c <- env(empty_env(), d = 4, e = 5)
e2d <- env(e2c, a = 1, b = 2, c = 3)

The ancestors of every environment eventually terminate with the empty environment. You can see all ancestors with env_parents():

env_parents(e2b)
#> [[1]] <env:   0x7fba74178f48>
#> [[2]] $ <env: global>
env_parents(e2d) 
#> [[1]] <env:   0x7fba74f70088>
#> [[2]] $ <env: empty>
</env:>

By default, env_parents() stops when it gets to the global environment. This is useful because the ancestors of the global environment include every attached package, which you can see if you override the default behaviour as below. We’ll come back to these environments in Section 7.4.1.

env_parents(e2b, last = empty_env())
#> [[1]]    <env: 0x7fba74178f48>
#> [[2]] $  <env: global>
#> [[3]] $  <env: package:rlang>
#> [[4]] $  <env: package:stats>
#> [[5]] $  <env: package:graphics>
#> [[6]] $  <env: package:grDevices>
#> [[7]] $  <env: package:utils>
#> [[8]] $  <env: package:datasets>
#> [[9]] $  <env: package:methods>
#> [[10]] $ <env: Autoloads>
#> [[11]] $ <env: package:base>
#> [[12]] $ <env: empty>

The ancestors of an environment have an important relationship to <<-. Regular assignment, <-, always creates a variable in the current environment. Super assignment, <<-, never creates a variable in the current environment, but instead modifies an existing variable found in a parent environment.

x <- 0
f <- function() {
  x <<- 1
}
f()
x
#> [1] 1

If <<- doesn’t find an existing variable, it will create one in the global environment. This is usually undesirable, because global variables introduce non-obvious dependencies between functions. <<- is most often used in conjunction with a function factory, as described in Section 10.2.4.

You can get and set elements of an environment with $ and [[ in the same way as a list:

e3 <- env(x = 1, y = 2)
e3$x
#> [1] 1
e3$z <- 3
e3[["z"]]
#> [1] 3

But you can’t use [[ with numeric indices, and you can’t use [:

e3[[1]]
#> Error in e3[[1]]: wrong arguments for subsetting an environment

e3[c("x", "y")]
#> Error in e3[c("x", "y")]: object of type 'environment' is not
#> subsettable

$ and [[ will return NULL if the binding doesn’t exist. Use env_get() if you want an error:

e3$xyz
#> NULL

env_get(e3, "xyz")
#> Error in env_get(e3, "xyz"): object 'xyz' not found

If you want to use a default value if the binding doesn’t exist, you can use the default argument.

env_get(e3, "xyz", default = NA)
#> [1] NA

There are two other ways to add bindings to an environment:

• env_poke()¹ takes a name (as string) and a value:

  env_poke(e3, "a", 100)
  e3$a
  #> [1] 100
  

• env_bind() allows you to bind multiple values:

  env_bind(e3, a = 10, b = 20)
  env_names(e3)
  #> [1] "x" "y" "z" "a" "b"
  

  You can determine if an environment has a binding with env_has():

  env_has(e3, "a")
  #>    a
  #> TRUE
  

Unlike lists, setting an element to NULL does not remove it, because sometimes you want a name that refers to NULL. Instead, use env_unbind():

e3$a <- NULL
env_has(e3, "a")
#>    a
#> TRUE

env_unbind(e3, "a")
env_has(e3, "a")
#>     a
#> FALSE

Unbinding a name doesn’t delete the object. That’s the job of the garbage collector, which automatically removes objects with no names binding to them. This process is described in more detail in Section 2.6.

There are two more exotic variants of env_bind():

• env_bind_lazy() creates delayed bindings, which are evaluated the first time they are accessed. Behind the scenes, delayed bindings create promises, so behave in the same way as function arguments.

  env_bind_lazy(current_env(), b = {Sys.sleep(1); 1})
  
  system.time(print(b))
  #> [1] 1
  #>    user system elapsed
  #>   0.001 0.000 1.003
  system.time(print(b))
  #> [1] 1
  #>    user system elapsed
  #>       0       0      0
  

  The primary use of delayed bindings is in autoload(), which allows R packages to provide datasets that behave like they are loaded in memory, even though they’re only loaded from disk when needed.

• env_bind_active() creates active bindings which are re-computed every time they’re accessed:

  env_bind_active(current_env(), z1 = function(val) runif(1))
  
  z1
  #> [1] 0.0808
  z1
  #> [1] 0.834
  

  Active bindings are used to implement R6’s active fields, which you’ll learn about in Section 14.3.2.

1. List three ways in which an environment differs from a list.

2. Create an environment as illustrated by this picture.

   

3. Create a pair of environments as illustrated by this picture.

   

4. Explain why e[[1]] and e[c(”a”, ”b”)] don’t make sense when e is an environment.

5. Create a version of env_poke() that will only bind new names, never re-bind old names. Some programming languages only do this, and are known as single assignment languages (http://en.wikipedia.org/wiki/Assignment_(computer_science)#Single_assignment).

6. What does this function do? How does it differ from <<- and why might you prefer it?

   rebind <- function(name, value, env = caller_env()) {
     if (identical(env, empty_env())) {
       stop("Can't find `", name, "`", call. = FALSE)
     } else if (env_has(env, name)) {
       env_poke(env, name, value)
     } else {
       rebind(name, value, env_parent(env))
     }
   }
   rebind("a", 10)
   #> Error: Can't find `a`
   a <- 5
   rebind("a", 10)
   a
   #> [1] 10